CO-MOT: Boosting End-to-end Transformer-based Multi-Object Tracking via Coopetition Label Assignment and Shadow Sets

We thank the reviewer gEi4 for the insightful comments and suggestions. We detailedly respond to each comment in the following.

Response to Weaknesses 1:

1.1 Due to the limited time of the rebuttal, we will supplement the results adding synthetic dataset.

1.2 MOTRv2 is transformer-based, but due to the addition of YOLOX preprocessing, it has a larger computational load and is not a pure End-to-end method. The original intention of this paper is to maintain the End-to-end structure and computational load of MOTR and achieve performance comparable to MOTRv2. The strong performance in Table 2 has well supported our motivation on multiple datasets.

1.3 The table below lists the performance of known End-to-end methods on the MOT17 test. It can be seen that End-to-end tracking has been prevalent in the past several years, and its performance has gradually improved. Currently, End-to-end MOT methods are still in the early stages of development, and we believe that there will be significant improvements in the future. The value of a method should not be denied just because it has not achieved state-of-the-art performance on a particular dataset. For example, although DETR and Deformable DETR have lower performance on the COCO dataset compared to other non-Transformer detection methods, their contributions to subsequent methods cannot be ignored.

Table 2 (c): Comparison to End-to-end methods on the MOT17 test dataset. The methods with bold italic fonts are newly introduced. CO-MOT has achieved the highest HOTA(60.1). Although the HOTA performance of MOTRv3 is the same as CO-MOT, it uses a more powerful backbone (ConvNeXt-B).

Response to Weaknesses 2:

The following table studies the performance impact of COLA inserting different decoder layers. l<0 means that COLA is not used, and all 6 layers of the decoder use TALA. l<2 means that the first layer of the 6-layer decoder uses COLA. It can be seen that deploying COLA in more decoder layers leads to better HOTA. As shown in Figure 3, during training in MOTR, TALA forcibly matches the tracking query with the corresponding track object, even if the current prediction of the tracking query is far from the target. This strategy makes it more difficult for the network to converge compared to the optimal solution of the Hungarian matching, which in turn makes it difficult for the track query to reach the optimal solution. This can also be seen in Figure 1 in the motivation section. Introducing COLA can improve this convergence. The detection query recalls the tracked targets through Hungarian matching and then passes the relevant information to the track query through the attention mechanism of the Transformer. With the prior knowledge from the detection query, the convergence of the track query becomes easier, and the target will be consistently assigned the same identity.

Table 8: The performance metrics of COLA inserting different numbers of decoder layers.

Response to Weaknesses 3:

Thank the reviewer for the provided advice to make this work more technically solid. "min" and "max" used to select the samples for training are heuristic but effective. More exploration such as importance sampling for sample selection will be conducted in our future work.

Response to Question 1:

The following two tables respectively show the ablation experiments of inserting COLA and Shadow set into TrackFormer and MOTR. As components are added, the tracking performance gradually improves.

Table 4: The ablation experiments of applying CO-MOT to TrackFormer on the DanceTrack validation set.

Table 9: The ablation experiments of applying CO-MOT to MOTR on the DanceTrack validation set and MOT17 test.

Thanks for the author for the feedback. I have carefully read the rebuttals.

1. Regarding "End-to-end MOT methods are still in the early stages of development, and we believe that there will be significant improvements in the future. The value of a method should not be denied just because it has not achieved state-of-the-art performance on a particular dataset..." I do agree that not achieving SOTA should not be the reason to reject a submission. But, DETR is the first work to make Transformers work for object detection and thus is a $**0-1**$ work, so no one will criticize its inferior performance compared to non-Transformer based detectors and it has influenced lots of follow-up works. However, the current submission is nothing like DETR, as lots of Transformer based trackers have already been published, e.g. TrackFormer/MOTR/MOTRv2, therefore the current work is not $**0-1**$ work like DETR but $**1-2, 2-3**$ $**incremental**$ work, it is just about "climbing mountains" yet the performance is not even better than non-Transformer based trackers, though I'm not criticizing it not being SOTA. Making it impossible to justify the motivation as, sadly I didn't see novel insights nor strong results.

2. I do agree with the argument that " More exploration such as importance sampling for sample selection will be conducted in our future work". I hope the author could improve towards a better formulation for this part in the future work.

I think for conferences like NeurIPs/ICLR, we should have more expectations for submissions. I therefore continue with my rating and I hope author would continue working on improving it towards a stronger submission.

We thank the reviewer oLii for the insightful comments and suggestions. We detailedly respond to each comment in the following.

Response to Weaknesses 1:

In the multi-object tracking (MOT) task, it is necessary to keep tracking the targets that are tracked in the previous frame, while also detecting newborns that appear in the current frame. The tracking queries and detection queries play different roles in these tasks. During training, they are supervised by different supervision, where the detection queries use Hungarian matching, and the tracking queries use TALA. During inference, the tracking queries and detection queries are concatenated together and indiscriminately passed through multiple Transformer decoders. Thus, all the queries can perceive the whole image, as the reviewer points out, but both tracking and detection queries must exist for MOT.

Response to Weaknesses 2:

As the End-to-End solution has just emerged in the past year, there are not many methods evaluated on MOT20 that we could find. Here are the ones in Table 2 (d). Notably, our approach achieves 57.5% HOTA, which is the state-of-the-art in End-to-end tracking methods.

Table 2 (d): Comparison to existing methods on the MOT20 test dataset.

Response to Weaknesses 3:

The following table lists the mAP indicators, which are consistent with the DetA indicators in Table 2(a). The detection performance of CO-MOT is far superior to MOTR but slightly inferior to MOTRv2.

Tracking tasks not only include detecting new targets but also need to track targets that have already been tracked, maintaining the identity of the targets between consecutive frames. Therefore, CO-MOT cannot be used only as a detection task, otherwise, it would not be able to associate targets between previous and subsequent frames.

Table 6: The detection performance (mAP) of MOTR (v2) and CO-MOT on DanceTrack validation dataset.

Response to Weaknesses 4:

We ablate the number of decoders and illustrate the results in Table 7. The results show that the HOTA consistently improves as the number of decoders increases, while other metrics significantly increase or are comparable. To make a fair comparison with MOTR and consider the efficiency, we choose six decoders by default in all the experiments.

Table 7: Ablation study on the number of decoders L. As L increases, the computational load increases, and the HOTA tracking performance improves, but the growth rate of HOTA significantly decreases.

Response to Weaknesses 5:

We reorganize the formats of the references according to the official regulations in the manuscript.

We thank the reviewer NAsG for the insightful comments and suggestions. We detailedly respond to each comment in the following.

Response to Weaknesses 1:

In the supplementary materials, we provide the results of COLA and Shadow Set on Trackformer. We do not make any modifications to the TrackFormer framework and used the default hyperparameters provided by TrackFormer. The result shown as follows demonstrates the effectiveness of COLA (+6.4%) and Shadow Set (+9.3%) when applied to TrackFormer. Codes will be released upon acceptance.

Table 4: The ablation experiments of applying CO-MOT to TrackFormer on the DanceTrack validation set. As components are added, the tracking performance improves gradually.

Response to Weaknesses 2:

The Figure 7 illustrates the failure cases of MOTR and denotes which kind of case has been solved by CO-MOT. MOTR has poor detection and tracking performance. First, as shown in Figure 7 (a), it fails to detect the person in time under the extreme case of bending over. Second, as shown in Figure 7 (b), due to the tiny visual features when a person stretches out their hand, the detection box is inaccurate, and the model misidentifies it as multiple people. Third, as shown in Figure 7 (c), the tracking identity switches after the human body is obscured. However, all of the above cases can be solved by CO-MOT, showing the extraordinary performance of MOT.

Response to Question 1:

End-to-end tracking [1,2,3,4] is defined as inputting contiguous video frames and directly outputting the boxes and association information without any pre- or post-processing (such as separate detectors, Reid feature extraction, or IOU match). So methods such as MOTRv2, ByteTrack, TransTrack, and OC-SORT belong to Non-End-to-end tracking. On the contrary, MOTR, MeMOTR, MeMOT does indeed belong to an end-to-end approach. We append more Transformer-based methods for comparison on MOT17 in Table 2 (c). The methods with bold italic fonts are newly introduced.

Table 2 (c): Comparison to existing methods using Transformer on the MOT17 test dataset. The last few lines are all end-to-end tracking methods, among which CO-MOT has achieved the highest HOTA. Although the HOTA performance of MOTRv3 is the same as CO-MOT, it uses a more powerful backbone (ConvNeXt-B).

Response to Question 2:

We show the specific numbers in the following table and indicate them in Figure 4.

Table 5: The parameters, model size, and FLOPs of MOTR, MOTRv2, and CO-MOT.

References:

[1] F. Zeng et.al. Motr: End-to-end multiple-object tracking with transformer. ECCV 2022

[2] Y. Zhang et.al. Motrv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors. CVPR2023

[3] J. Cai et.al. Memot: multi-object tracking with memory. CVPR2022

[4] T. Meinhardt et.al. Trackformer: Multi-object tracking with transformers. CVPR2022

[5] Yu et.al. MOTRv3: Release-Fetch Supervision for End-to-End Multi-Object Tracking. https://arxiv.org/abs/2305.14298

We thank the reviewer kcLe for the insightful comments and suggestions. We detailedly respond to each comment in the following.

Response to Weaknesses 1:

Thank you for your constructive feedback on the paper organization. We have updated the manuscript according to your suggestions to improve its readability and comprehensibility. For example, we move Figure 1 near the method section so that readers can refer to it more conveniently. Additionally, we demonstrate the symbols of Section 3.2 in Figure 2. To highlight the technical improvements of the new approach, we add a detailed figure of MOTR (v2) and CO-MOT in Figure 6. Note that the stacked circles and squares denote the shadow sets.

Response to Weaknesses 2:

We demonstrate tracking methods as many as possible in the supplementary materials, which are evaluated on DanceTrack. The methods with bold italic fonts are newly introduced. For BDD100K, we have already shown all the methods evaluated in Table 2(b) so that no more methods are introduced.

Table 2 (a): Comparison to existing methods on the DanceTrack test set. The last few lines are all end-to-end tracking methods, among which CO-MOT has achieved the highest HOTA.