Learning Pseudo 3D Representation for Ego-centric 2D Multiple Object Tracking

We sincerely thank you for the professional comments. We are inspired and hope our discussion brings more insights.

W1: Ablation study on KITTI

• We conducted ablation study on Waymo open dataset, because it is a more large-scale dataset than KITTI. Besides, there is no validation set on KITTI. The previous works that conducted ablation study on KITTI split each training sequence into two halves, and use the first half frames for training and the second for validation. The overlapping videos cause inaccurate (too high) results that cannot reflect the actual performance well.
• We also supplement the ablation study on KITTI.

W2: More qualitative results

We show the qualitative results (videos with tracked objects) in supplementary material zip file. To show more results, we also build a project page https://p3dtrack.github.io/.

W3: Inference time

• We measure the latency on NVIDIA A40 GPU, the same as it was trained on. We use a single GPU to measure the latency.
• The inference latency is mainly dominated by the object detector, for about 150ms per frame. The tracking stage (feature extraction and object association) takes about 80 ms. If using a real-time object detector, like the speed-optimized YOLO-series detector (YOLOX_x 17.3 ms, YOLOv8x 3.53ms), we believe it will be a (near) real-time system for autonomous driving (sensor input is 10Hz for most autonomous driving systems). Besides, please note the 3D reconstruction is only conducted in pseudo-label generation stage, and it will not affect the inference speed.

Q1: Object detector

CenterNet is a commonly used detector in MOT, so we chose it as our base detector for a fair comparison with SOTA methods. (Because the detector is not the core concern in our method.)

We also conduct additional experiments on a more modern detector to show the performance with a better detector. We show the results with the YOLOX detector (also widely used recently, especially for ByteTrack-series trackers.) The detection performance is very well on Waymo open dataset.

Based on it, we can achieve better tracking results.

Q2: At night

When the visibility is low at night, the SfM system will be negatively affected. However, it only slightly affects pseudo label generation. The network learning 2D boxes and 3D representation is less affected. We show a video on the anonymous project page.

We are grateful for your valuable comments and constructive advice, which help us a lot to make the paper better.

incremental improvements over SOTA

The main reason is that we use the weak CenterNet-based object detector with DLA-34 backbone, even worse than FasterRCNN_R50 in QDTrack. On COCO, there is a 1~2AP detection performance gap (39.2 vs 37.4). (We use CenterNet because a single-stage object detector makes it easier to add a 3D representation head.) However, we also show the performance increase with object detector scaling-up. When using a better detector YOLOX to compensate for the detector performance gap, we can outperformance SOTA methods by a large margin.

Failure cases

The common failure case of our method is the occluded scene, like parking lot, distant objects, occluded objects very close to the camera. The 3D location for these cases in naturally hard to estimate. The detailed qualitative results can be found on the anonymous project page: https://p3dtrack.github.io/.

Non-static object

Firstly, we show the object-level depth estimation results between car and pedestrian.

The results show that even though pedestrians are more likely to be dynamic, the results for pedestrian are still no worse than vehicle. The object tracking results for these two classes are shown in the following Table.

Then we discuss more about why there is no significant static bias. Our method for 3D representation learning contains two main parts: 3D pseudo-label generation and 3DRL with 3D pseudo label.

• As for 3D pseudo-label generation, "the points on dynamic objects are mostly ignored". That means 3D pseudo labels are rarely generated for pedestrians due to their movement. (However, when they are static, such as waiting for a red light, the pseudo label can still be generated.) This is the natural shortage of SfM.
• Although lacking 3D pseudo labels, we can utilize the "generalization ability of depth from other objects" of the object 3D representation learning module. The other objects are mainly vehicles that are learned with 3D pseudo labels. We can generalize the depth of pedestrians from the learned depth of vehicles. This generalization ability is because depth is the class-agnostic attribute of the object, and the network learns depth from the whole image. During the inference stage, the network can leverage the learned depth information of other vehicles in the same image to predict the depth of pedestrians.

Thanks for your comment. However, we kindly hope the reviewer can read our paper again, because some comments deviate from the facts.

Response to the comments in Strengths:

• Why talk about LiDAR-based 3D MOT and why cite Kalman Fiter in the introduction?

We observed that using only the Kalman filter works very well for LiDAR-based 3D MOT. Therefore, we hope to help 2D MOT with a 3D representation.

W1: The paper does not really compare to state of the art methods for MOT.

We tried our best to compare the SOTA methods. As for KITTI dataset, please refer to https://www.cvlibs.net/datasets/kitti/eval_tracking.php for the tracking leaderboard. All published image-based SOTA methods are compared in Table 2. For Waymo Open Dataset, we compared our method with real SOTA QDTrack. For other methods not reported on these two datasets, such as ByteTrack, we implemented it using the official code and reported it in our ablation study (Table 3 Row 2). We think we have sufficiently compared the existing SOTA MOT methods.

W2: Also, the datasets chosen do not really demonstrate the ability to do multiple object tracking as at most there is 2 objects being tracked.

It seems the reviewer is not familiar with these datasets. As the widely-used MOT datasets, KITTI and Waymo have far more than 2 objects being tracked. More than 100 methods report MOT results on these datasets. We believe that these datasets can demonstrate the capability of MOT.

W3: With regards to MOT, the datasets chosen should have been from the MOT challenge.

As mentioned in Section 4.1, we discussed why we don't choose MOT17 and MOT20 as the evaluation dataset.

Please note that other datasets, such as MOT17 and MOT20, include numerous surveillance videos without camera motion. These datasets are not suitable for evaluating our method.

SfM cannot be operated in non-geo-centric videos. Ego-centric video means the camera is moving along with the ego.

W4: The authors claim that using 3d features simplifies the problem, which it does for the dataset used, as objects are clearly separated by depth.

Please note that as a pure image-based MOT method, objects cannot be clearly separated by depth, how we separate them without any depth labels is indeed our contribution. Besides, pretrained depth cannot beat us. (Table 5)

W5: In addition, datasets that should have been considered also include GMOT-40 and Animal Track.

The GMOT-40 and Animal Track datasets have almost no necessary connection with our approach and our ego-centric setting. And compared with the datasets we used, GMOT-40 and Animal Track are much less popular.

W6: Long term pixel tracking is of interest recently which makes sense when objects are not separated by depth as in the examples given.

We appreciate your advice, but we don't know the necessity of using long term pixel tracking methods in our paper for now. Pixel tracking and object tracking are totally different tasks. We agree that it may help to track objects, however, there is no connection with our pseudo 3D representation.

Q1: Is this more than just a simple tracking by detection?

As shown in the introduction, we summarize the contribution of the paper.

(1) We propose a new online 2D MOT paradigm, called P3DTrack. P3DTrack utilizes the jointly learned 3D object representation and object association. (2) We design the 3D object representation learning module, called 3DRL. With 3DRL, the object’s 3D representation is learned from the video and supervised by the 2D tracking labels without any additional annotations from LiDAR or pretrained depth estimator. (3) The extensive experiments on the large-scale ego-centric datasets, KITTI and Waymo Open Dataset (WOD) show that we achieve new state-of-the-art performance.

The most important contribution lies in the new pseudo 3D representation without depth/LiDAR supervision.

Q2: You do not appear to do a predict and correct by detection or observation as would be necessitated by a kaman filter.

As mentioned in Section 3.3, the proposed tracker infers similar to DeepSORT. In DeepSORT, the tracker predicts and corrects the tracklet by detection. Since we did not specifically mention, we don't change this process.

Q3: IF you are using a Kalman filter, this is linear and will fail when motion is not linear?

No. As discussed in many MOT papers, Kalman Filter can work for regular high-frame-rate videos.

Q4: For all your examples, the objects are well separated by depth, what happens when they are not?

As mentioned in Section 3.2, besides 3D representation, we also have appearance model just like other papers. When 3D representation fails, the appearance model can provide available association.

In summary, it appears that there are many misunderstandings in comments regarding our work and the field of MOT. Please consider reading related work and our paper again.

We really appreciate your valuable comments. We are trying our best to address your concerns and are open to any further discussions.

W1: Limited novelty

• Our main novelty and contribution lie in the 3D representation. Specifically, We focus on how to obtain 3D labels, how to generalize the label to other objects, and how to use the (not so accurate) 3D features in MOT. The experiments also show that our 3D representation yields a substantial improvement compared to traditional 2D representation and other pretrained 3D features. As for technical contribution, to obtain 3D labels, we design the two-stage object clustering (IntraPC and InterPC). To generalize the label to other objects, we propose an image-based 3DRL module learning from all 2D labels and static 3D labels, and verify that this learning module can generalize the static labels to dynamic objects. To measure the degree of inaccuracy, we introduce uncertainty-based 3D loss. We believe this learning paradigm can inspire the MOT community to pay attention to mining the 3D information hidden in the video that does not require manual annotation.
• The feature aggregation and association modules are not the main contribution in our paper. For the online "Tracking by Detection" paradigm, most methods share a very similar association module, i.e., encoding the appearance/motion features, aggregating the object features, matching between detections and tracklets. This pipeline design can even back to DeepSORT published in 2017, and further adopted and slightly modified by the following work GMTracker [A], ByteTrack [B], BoT-SORT [C], etc. In this paper, we only utilize the best practices in the data association stage to obtain better performance and do not focus on the novel design in this part.
• About the details in 3D pseudo label generation, we try our best to solve your concern in the following.

W2: Limited impact

• We agree that most existing ego-centric datasets have LiDAR sensors. However, with the development of embodied AI and vision-based autonomous driving, hand-held cameras and cameras on vehicles without LiDAR will be widely used. For example, using DJI drones, driving Tesla autonomous-driving vehicles, and biking with a GoPro camera will provide more available data. We believe our algorithm will have great potential in these fields.
• Besides, we also report the additional results on the general non-ego-centric dataset MOT17 test set. The results show that even though training 3D representation on only half of the videos that have ego-motion, we still obtain performance improvement, especially compared with pretrained depth-based tracker QuoVadis [D]. Please note in this Table, we change our base detector to YOLOX to align the detector performance with SOTA methods.

[A] He et al. Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking. In CVPR 2021.

[B] Zhang et al. ByteTrack: Multi-Object Tracking by Associating Every Detection Box. In ECCV 2022.

[C] Aharon et al. BoT-SORT: Robust Associations Multi-Pedestrian Tracking. arXiv 2206.14651.

[D] Dendorfer et al. Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking? In NeurIPS 2022.

Q1: Filtering

• We filter low speed of ego-motion videos according to the ego speed from GPS/IMU. The videos in which the ego-vehicle moves less than 20m will not be considered in the reconstruction. In Waymo open dataset, this threshold can easily separate low-speed and normal videos.
• We filter the moving objects by the threshold of reconstructed 3D points. Moving objects have fewer reconstructed 3D points because the movement is not consistent with ego and cannot be reconstructed with many 3D points. The threshold is 30 3D points. Typically, there are more than 30 keypoints extracted from the object in one frame. If this threshold is not met, it indicates that there aren't enough matched keypoints between two frames for this object, suggesting that it may be a moving object.
• About how moving objects are being handled: (1) As for 3D pseudo-label generation, "the points on dynamic objects are mostly ignored". That means 3D pseudo labels are rarely generated for moving objects, including moving vehicles and most pedestrians. (However, when they are static, such as a vehicle/pedestrian waiting for a red light, the pseudo label can still be generated.) This is the natural shortage of SfM. (2) Although the 3D representation label is not generated for moving objects, we can generalize the 3D representation from the static objects. This is because the 3DRL module learns object's 3D representation from a single image instead of a video. In the image, network predicts the 3D representation from appearance instead of motion, so moving and static objects are not different. Even though we only learn from static objects, the object's depth can be generalized to other moving objects. In detail, we can utilize the "generalization ability of depth from static objects" of the object 3D representation learning module. This generalization ability is because depth is the class-agnostic attribute of the object.

Q2: Learning from static object

Just as mentioned in response to Q1, the 3DRL module learns object's 3D representation from a single image. Like monocular 3D object detector, on the head of CenterNet, we regress the object's 3D position ($H\times W\times 3$) from $H\times W$ feature map. However, the supervision is only on the 2D center of static objects. As mentioned in response to Q1, we learn class-agnostic 3D representation from monocular image, and in an image, moving and static objects are not different in appearance. Moving and static objects share the same prediction head, so $o^t_j$ will not be any values for moving objects.

Q3: Impact of pseudo label quality

We conduct additional experiments using real 3D object labels to show the impact of pseudo label quality. Please note that, in Waymo open dataset, real 3D object labels are from LiDAR (0-75m, VFOV [-17.6, +2.4)), which means 3D labels are fewer than 2D labels. Besides, the 3D object ID GT and 2D object ID GT are not aligned (the same object has a 3D ID and a 2D ID). So, we match the 3D GT and 2D GT using the maximum IoU matching. We also retrain our model to fit the objects in this subset.

Q4: Increase of ID Sw

• It is because in the last row, the object recall is much higher than the baseline. Please refer to the FP and FN metrics in Table 3. ID Switches can only be compared under similar recall performance. In general, the more objects considered in tracking, the more ID Switches are reported. Besides, IDF1 is also the metric to reflect the identification performance. We obtain 5.8 IDF1 improvement (68.1 vs. 62.3).
• 3D representation can help reduce ID Sw. Please refer to Table 4. In this Table, we only change the 2D/3D representation without modifying other modules and thus maintain a similar recall, so it can reflect the influence of our 3D representation better.