CO-MOT: Boosting End-to-end Transformer-based Multi-Object Tracking via Coopetition Label Assignment and Shadow Sets

We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments.

W1 and Q1: While an increase in the number of queries typically raises training and inference costs, the sampling framework used in our paper is similar to that of DETR, consisting of three main modules: ResNet for image feature extraction, the Encoder module for further integration of image features, and Decoder module for outputting bounding boxes and confidence scores. The increase in queries primarily affects the computation in the decoder layer; however, the decoder contains only six attention layers, which constitute a small portion of the overall model. As shown in the figure, the impact on inference speed is negligible (about 6%). The table lists the inference speeds and decoder FLOPs for different query configurations as follows:

W2 and Q2: We note that MOT20 indeed has a higher object density, which provides a more challenging environment for evaluating the effectiveness of the model.

This paper primarily focuses on end-to-end tracking methods. Currently, we have only found evaluations of MeMOT and TrackFormer on the MOT20 benchmark, which we have included in Table 3C. At the same time, we conducted a detailed performance analysis on three commonly used benchmarks: DanceTrack, BDD100K, and MOT17. Additionally, Table 1 lists the mAP of various methods, showing that CO-MOT significantly improves the recall of detection boxes.

It is worth mentioning that the phenomena observed in MOT20 and MOT17 are quite similar, with our method outperforming other end-to-end approaches across various metrics. For instance, in the HOTA metric, CO-MOT exceeds MeMOT by 3.4% and TrackFormer by 2.8%. These results indicate that CO-MOT demonstrates consistent performance across different datasets.

W3: We discuss these two questions in Table 9 and Table 5. The optimal number of shadow sets is found to be 3; fewer sets do not contribute effectively, while more sets can introduce negative side effects due to excessive variance within the same set. Additionally, COLA performs best within the l<5 decoders.

W4: Thank you once again for your valuable feedback. We will make the necessary modifications and adjustments based on your suggestions.

Thank you again for your time and efforts! We show our deepest appreciation for your support of our work. We are always ready to answer your further questions!

We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments.

W1: We highly value the feedback from the reviewers and AC, and we have made comprehensive improvements in the latest version, including restructuring the paper and adding a significant amount of experimental data, such as more datasets and updated ablation studies.

The ICLR2024 reviewers raised the following issues: see https://openreview.net/forum?id=WLgbjzKJkk. Below, we clarify the points that were raised in previous reviews one by one:

a: presentation issues, need to refer to baseline MOTR paper. We have significantly reorganized the structure of the entire paper and added relevant citations in the revisions. Additionally, the distance between figures and text has been improved to enhance readability and understanding, for example, by placing Figure 2 close to the corresponding text.

b: Needs more comparison on BDD100k and DanceTrack. We have added many comparative experiments in the BDD100k and DanceTrack evaluations, as detailed in Tables 2 and 3. These experimental results further validate the effectiveness of our method.

c: Missing evidence that COLA/Shadows works on other models like Trackformer. We have supplemented the ablation studies on Trackformer, as shown in Table 6. Additionally, COLA and Shadows Set have been introduced into the more powerful backbone, MeMOTR, resulting in performance improvements.

d: Provide failures cases specific to previous approaches that were solved by CO-MOT. Figure 8 lists specific cases that CO-MOT has resolved, showcasing the advantages of our method.

e: Table 2 presentation issues. We have introduced the definitions of non-end-to-end and end-to-end approaches in the introduction for better reader understanding.

f: Fig 4 show the number of parameters, FLOPS of YOLOX included in MOTR? Figure 4 displays the number of parameters to provide readers with a clearer understanding of the model's complexity.

g: Is it necessary to split tracking/detection queries? This issue arises from a misunderstanding of the paper. Tracking queries and detection queries are interdependent in our method and cannot be simply separated.

h: Missing evaluation on MOT20. We have included the experimental results for MOT20 in Table 3(c).

i: What if the tracking queries are removed in CO-MOT. Does detection result improve similar to MOTR? In Table 1(f), we have added the mAP data for CO-MOT, which shows that the detection performance of CO-MOT significantly exceeds that of MOTR.

j: effect of the number of decoders L on tracking performance. We have added relevant experiments in Table 5 to explore the impact of the number of decoders on tracking performance.

k: Performance improvement not consistent across different datasets. Ablation experiments based on multiple baselines across various datasets have been conducted and are presented in Tables 4, 6, and 7, all showing improvements.

l: Performance worse than MOTRv2 in terms of IDF1 and HOTA. We provided a detailed explanation in Section 3.4 (COLA) and Figure 3 regarding this phenomenon.

m: Lack of interpretability of proposed method. We have included extensive explanations and data analysis in Section 3.3 (MOT7) to enhance the interpretability of our method.

n: Lack of mathematical formulation for shadow sets -- too many engineering tricks or heuristics. Detailed explanations and data analysis regarding the design principles of shadow sets are provided in Section 3.4.

o: missing ablation study on w/ and w/o COLA/shadow set on MOT17 validation set. We have supplemented the relevant experiments in Table 7.

We believe that these revisions have significantly improved the quality and contributions of the paper.

W2: These methods (such as DiffMOT, MambaTrack, TrackSSM, et al) demonstrate excellent performance, indicating that end-to-end tracking approaches are gaining increasing attention. We provide a comparison with our method, CO-MOT, as shown below:

Our CO-MOT method outperforms the aforementioned methods across multiple key metrics, demonstrating its effectiveness and advantages in the target tracking task. This result further emphasizes the innovation and potential of our approach.

Thank you again for your time and efforts! We show our deepest appreciation for your support of our work. We are always ready to answer your further questions!

I have carefully read the rebuttal and thank the author for their work. I have compared the differences from last year, and found that the author has made many improvements in both the experiment and writing. While the experimental results are highly impressive on DanceTrack, this work exhibits incremental novelty. I have decided to improve the rating to 6.

We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments.

W1: The authors should provide a detailed discussion of the differences between COLA and TALA in Section 2.4, as well as their design in the loss function, to facilitate reader understanding.

In Figure 6, we briefly explain the differences between COLA and TALA. As shown in Figure 6(a), in COLA, the detection queries can only match with newborns 2 and 1; whereas in TALA, the detection queries can match not only with newborns 2 and 1 but also with tracked individuals 3 and 4. To provide a more detailed description, we will set aside a separate paragraph to discuss the differences between COLA and TALA in depth.

W2: In the experiments section, the authors need to include comparisons with more methods on the MOT20 and BDD100K datasets.

In this study, we primarily focus on end-to-end tracking methods and have listed several comparative methods on the DanceTrack and MOT7 datasets.

Due to the relative novelty of our approach, there is currently limited research using MOT20 and BDD100K as evaluation benchmarks. Therefore, we have compiled all known end-to-end tracking methods in the table to provide clear references for readers.

We will continue to monitor developments in this field and consider incorporating more comparisons in future work.

W3:Since the authors analyze the impact of tracking queries on detection performance in transformer-based trackers, if this point serves as one of the motivations, they should compare whether the proposed framework shows improvement in mAP in the experiments.

In Table 1, we present the mAP results of representative methods. The mAP of CO-MOT is significantly higher than that of MOTR and slightly lower than that of MOTRv2, indicating that our method is competitive in performance.

W4: The authors should also analyze the effects of different values of λ and Φ in Section 2.5 on the experimental outcomes.

We have conducted numerous relevant experiments in Table 8 to explore the effects of different λ and Φ values on the experimental results.

Thank you again for your time and efforts! We show our deepest appreciation for your support of our work. We are always ready to answer your further questions!

We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments.

W1: To solve the issue of disproportional assignment of track and detection queries, there are also other simpler alternatives. A straightforward option would be to train detection queries jointly on image detection datasets alongside video tracking datasets. For example, detection queries could be trained exclusively on image datasets, treating every object as a new object. An ablation study comparing proposed methods to this simple joint training alternative is appreciated.

Yes. For instance, MOTRv2 uses a pre-trained YOLOX to extract detection boxes, significantly enhancing tracking performance. In Table 2, CO-MOT+ uses a combination of Crowdhuman and video data for joint training, which further improves the tracking results on DanceTrack. Therefore, it can be confidently stated that adding a substantial amount of image datasets can indeed enhance tracking performance.

W2: The paper uses the first 5 decoders to train with all queries, while the last one trains separately on detection and tracking queries. An ablation study could clarify whether a different arrangement, such as using the first decoder for track queries and the last five for all queries, would impact performance. An ablation study regarding this would be helpful for readers to understand the optimal configuration.

Yes. As shown in Table 5, we have validated the effect of COLA across different decoders. The experiments confirm that using all queries simultaneously on the first five decoders yields the best results, but it is essential to ensure that the last decoder uses the detection and tracking targets obtained from COLA.

W3: The applicability of the coopetition label assignment strategy is mostly limited to cases where there is more video data than image data for training, leading to an imbalance in track and detection query assignments. However, in many practical settings, the opposite is true—large-scale [1] and open-vocabulary MOT tasks [2] often have substantially more image detection data than video tracking data. In these cases, common practice in MOT is to use joint training with both image and tracking data, which provides sufficient supervision for detection queries. This is contrary to the paper’s analysis, and it would be beneficial for the authors to also at least discuss these more common scenarios.

Yes. Annotating tracking video data, in particular, requires significant human and financial resources. In contrast, there is currently a wealth of image detection data available, which can be enhanced to further improve tracking performance. Many recent studies have adopted this approach, such as MOTRv2 and MOTRv3, and our CO-MOT+ also further validates this conclusion.

Q1: The main experiments are still concentrated on small-scale pedestrian tracking datasets. As mentioned on weakness, for other scenarios, we may face different difficulties. Are there any plans to test the model also on large-scale MOT datasets such as TAO [3]?

Thank you very much for your valuable suggestions. TAO is an excellent benchmark, but it is not well-suited for training models like MOTR and CO-MOT. We have previously conducted experiments on TAO, but the results were not ideal, mainly due to the large amount of unannotated data in TAO, which is more suitable for pre-training or open-vocabulary MOT tasks. However, we will continue to monitor developments in this field and explore more general MOT models.

Thank you again for your time and efforts! We show our deepest appreciation for your support of our work. We are always ready to answer your further questions!