S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking

Thanks for your time and insightful feedback. We especially appreciate your recognition of our well-designed modules with innovative improvements, and found our impressive performance validates the potential of end-to-end methods. We responded in detail below and will add them to the revision.

Q1: Discuss the computational complexity, in order to more clearly highlight the advantages and disadvantages of their method.

Thanks for your valuable suggestion! We provide the additional computational complexity of the proposed method. S2-Track only adds about 7.0% parameters, 2.4% FLOPs over PF-Track, while this trade-off results in a 12.3% improvement in AMOTA. Further improvements in tracking efficiency are a promising direction for future research. To highlight the disadvantages of our method, we have added a "Limitations and Future Work" section in the revision, hoping to inspire further exploration in this field.

Q2: Specify whether other methods use full resolution or reduced resolution

Thank you for your valuable suggestion! The numbers in our tables are directly taken from the corresponding methods' papers or the official nuScenes leaderboard, and we report the highest values for each method. Based on your suggestion, we have added a column to the tables in the revision to include the corresponding resolutions, allowing readers to make a more comprehensive assessment. Below, we provide some of the resolution settings.

Q3: If deployed on an in-vehicle GPU, would it meet the requirements for real-time inference? If not, how far is this end-to-end multi-object tracking approach from practical application

Thank you for your comment! We have successfully implemented our S2-Track on the real-world autonomous vehicle equipped with the NVIDIA Drive AGX Orin platform. By incorporating various engineering optimizations, such as TensorRT quantization, we have achieved real-time performance exceeding 20 Hz on the real-world vehicle. Additionally, we have provided a video showcasing the real-world results on an anonymous GitHub Page: https://anonymous-github-8ab1cv.github.io/s2-track/. However, to achieve large-scale production and deployment for high-level autonomous driving (e.g., L3/L4), further improvements in efficiency or platform (e.g., NVIDIA Drive AGX Thor) will be required.

Thanks again for your thoughtful feedback. We believe that your suggestions, along with our revision, have greatly enhanced the persuasiveness and completeness of our work. We hope our rebuttal can address your concerns.

Thanks for your time and insightful feedback. We especially appreciate your recognition of our three useful improvements and superior tracking performance. We responded in detail below and will add them to the revision.

Q1: How to initialize object queries with 3D location.

Thanks for your comment! We ulitze the 3D location to initialize queries by following steps: 1) normalize input coordinates to the [0, 2π] range; 2) generate frequency bands using exponential temperature scaling; 3) compute sine/cosine components for each dimension (X,Y,Z); 4) concatenate the encoded dimensions; 5) project the concatenated features through two linear layers with ReLU activation. We will include these implementations in the revision.

Q2: More explanations of core ideas in HQD.

Thanks for your suggestion! In complex 3D MOT scenarios, challenges such as occlusions and varying object sizes can hinder the learning and convergence of query-based methods. The slow convergence and suboptimal results from the instability of bipartite graph matching. To this end, we perturb GT bounding boxes with noises into the decoder and train the model to reconstruct the original boxes, which effectively reduces graph matching difficulty and leads to faster convergence. Moreover, we define hierarchical challenging levels for the perturbed queries to enhance the model’s ability to handle diverse driving scenarios. We have included these explanations in the revision.

Q3: How do you come up with these improvements? Are there connections among these ideas?

Thanks for your comment! As mentioned in the Introduction L71–101, with the goal of enhancing existing end-to-end tracker in complex driving environments, we first decompose current query-based framework into three constituent parts: query initialization, propagation, and matching (Fig. 1(b)). Then we propose corresponding improvements for each part: PQI, UPD, and HQD. These modules are connected by their shared foundation—the query-based framework, with all improvements targeting challenges posed by complex environments. As Reviewer WzGu acknowledged, "The three modules (PQI, UPD, HQD) address the tasks in different stages of the query tracking lifecycle."

Q4: all these improvements are just other papers’ ideas

Thanks for your comments! We, with respect, did not agree. Current end-to-end trackers are still in the early stages and struggle with complex driving scenarios. In response, S2Track comprehensively improves existing framework. For PQI module, we leverage predicted 2D locations and depth information to enhance the queries initialization. While previous works in detection have explored the use of depth information, none have leveraged it for query initialization. For UPD module, the uncertainty issue has never been explored in the 3D MOT, let alone propose an Uncertainty-aware Probabilistic Decoder for tracking. For HQD module, although it draws inspiration from DN-DETR, we have improved it by introducing Hierarchical Query Denoising. As demonstrated in the ablations (Tab 5), our improvements outperform the original DN-DETR. None of these proposed modules are just other papers’ ideas. Moreover, S2Track, is not a simple combination of these ideas; it delivers impressive tracking results, showcasing the potential of end-to-end framework, which are acknowledged by all other reviewers:

• Reviewer WzGu: "The design of UPD is novel... The framework is both simple and strong, .... It also achieves outperforming performance with the refined transformer query mechanisms."
• Reviewer WzGu: "... incorporating three novel modules, ... brought by the newly-designed modules."
• Reviewer 4TdZ: " (PQI&UPD) represent innovative improvements, (PQI) sounds intresting, (HQD) is a novel contribution .. providing a new solution, (framework) the core contributions...validates the potential of end-to-end methods."

Finally, we try to understand the reviewer's perspective. However, without S2Track decomposing the current framework into constituent parts, and proposing targeted improvements, it would be difficult for the community to grasp current limitations clearly. If these problems were so easily solvable in tracking, how could S2Track achieve such a significant improvement (+8.9% AMOTA) over previous SOTA? As Reviewer 4TdZ stated: "Effectively enhances the performance of the end-to-end multi-object tracking paradigm, which sounds quite impressive." We sincerely appreciate that the reviewer can recognize the substantial efforts behind our simple and strong framework.

Thanks again for your thoughtful feedback and time. We believe that your suggestions, along with our revision, have greatly enhanced the persuasiveness and completeness of our work. We hope our rebuttal can address your concerns.

Thanks for your time and insightful feedback. We especially appreciate your recognition of our effective framework with newly-designed modules and well-written paper. We responded in detail below and will add them to the revision.

Q1: The proposed method is only evaluated on nuScenes dataset

Thanks for your suggestion! First, since the nuScenes dataset provides comprehensive detection and tracking tasks, along with evaluation leaderboards, most previous detection and tracking methods (e.g., DQTrack [ICCV23], PF-Track [CVPR23], and ADA-Track [CVPR24]) have been only evaluated on nuScenes. Therefore, following them, we also perform a fair comparison on this dataset. Second, we agree with the reviewer’s point that methods evaluated on only one dataset have not demonstrated robustness across multiple datasets. To address this concern, we present additional evaluations on an in-house autonomous driving dataset, which is collected from real-world scenarios. The results show that our method effectively tracks objects in challenging environments, demonstrating the generalization and robustness of our S2-Track. Additionally, we have provided a video showcasing the real-world results on an anonymous GitHub Page: https://anonymous-github-8ab1cv.github.io/s2-track/.

Q2: The general framework is still based on existing methods. The insights to this task and the community is limited.

Thank you for your comments! However, we respectfully disagree with the "limited insights". While S2-Track employs the query-based end-to-end framework as previous works, as mentioned in the Introduction, current end-to-end trackers are still in the early stages of development and are unable to effectively handle the various complex driving scenarios and achieve satisfactory tracking results. Therefore, S2-Track comprehensively enhances the existing end-to-end 3D MOT framework, delivering impressive robust and accurate tracking results, and demonstrating the potential of end-to-end frameworks.

As Reviewer 4TdZ acknowledged, "The paper builds on recent advances in end-to-end query-based trackers and ... to address limitations in 3D MOT. Therefore, I believe one of the core contributions of this paper is that the performance of its proposed end-to-end multi-object tracking method surpasses both end-to-end and non-end-to-end methods. I think end-to-end approaches represent the future trend for both autonomous driving and multi-object tracking, and this paper validates the potential of end-to-end methods."

Q3: Analysis of other modules also effectively reduce the uncertainty.

Thanks for your comment! Our PQI module leverages learned certain priors, i.e., 2D object location and depth information, to enhance the initialization of queries, thus effectively reducing the uncertainty in query initialization and resulting in more accurate object localization and tracking. The HQD strategy introduces different levels of noise to the queries and then applies a denoising process, allowing the model to encounter varying magnitudes of noise (i.e., uncertainty) during training. This effectively helps the model reduce uncertainty during query matching, leading to more stable and accurate tracking performance. Although the motivation of these two modules is not uncertainty, they both help the model reduce uncertainty during query initialization and matching. Moreover, they are incorporated together with the UPD module, which aims to reduce uncertainty during query propagation. We will incorporate this discussion into the revision.

Q4: The parameters of HQD require searching.

Thanks for your comment! The HQD module indeed has two thresholds that need to be set. As shown in Tab 5, the impact of these two thresholds on the results remains relatively stable within a certain optimal range, meaning that we do not need to perform an extremely fine-grained search for their values. Simply identifying the approximate optimal range is sufficient. Furthermore, on our in-house autonomous driving dataset, we used the same values as those in nuScenes and also achieved satisfactory results. While a more detailed search might yield further improvements, the current settings already provide stable and satisfactory performance gains in most cases.

Q5: Add the GT in Fig 4.

Thanks for your suggestion! We have carefully included the ground truth as separate images in Fig4 in the revision, making it easier for readers to understand.

Thanks again for your thoughtful feedback. We believe that your suggestions, along with our revision, have greatly enhanced the persuasiveness and completeness of our work. We hope our rebuttal can address your concerns and sincerely appreciate that you can rejudge our efforts and potentially update your scores.

Thanks for your time and insightful feedback. We especially appreciate your recognition of our simple and strong framework with effective modules and SOTA tracking performance. We responded in detail below and will add them to the revision.

Q1: More analysis of other extreme scenarios

Thanks for your suggestion! In Table 4, we have already analyzed different occlusion situations, i.e., different visibilities. Here, we provide additional experiments of different weather and lighting conditions, and the results show that S2-Track is robust under different lighting and weather conditions, significantly boosting the performance under challenging rainy and nighttime scenes. The metric is AMOTA.

Q2: Results of category-aware AMOTA (e.g., pedestrian vs. vehicle)

Thanks for your valuable comment! We provide detailed category-aware AMOTA results on both the val and test set for better comparisons, as the previous methods or leaderboard reports comprehensive test set results. The results show that S2-Track achieves larger improvements in more challenging categories, e.g., pedestrian.

Q3: Other distributions for UPD

Thanks for your suggestion! We conducted additional experiments to explore different distributions. The results show that other distributions did not achieve satisfactory performance, which may be attributed to the natural statistical properties following the central limit theorem, i.e., many natural phenomena (e.g., lighting variations, sensor noise) arise from the superposition of numerous small and independent effects, leading to normal distribution.

Q4: Generalized to query-based 3D object detection.

Thanks for your suggestion! We have already presented our detection results on the nuScenes test and val sets in Table 9 and 10. As a framework designed for tracking, our model also achieves leading detection performance (62.7% mAP and 68.0% NDS on the test set), which clearly demonstrates our strong generalizability. While integrating our powerful modules into existing SOTA detection methods could lead to further improvements, due to computational resource constraints and the scope of this tracking paper, we leave this exploration to future work.

Q5: Missed reference: 3DPPE

Thanks for your comment! While 3DPPE [1] also involves depth priors in a query-based framework, it differs from S2-Track in several aspects. First, 3DPPE focuses on 3D object detection, whereas we tackle 3D MOT. Second, 3DPPE introduces 3D point positional encoding, while our PQI is designed for query initialization. Moreover, we also retain randomly initialized queries to explore missing objects. We will add this discussion into the revision.

[1] 3DPPE: 3D Point Positional Encoding for Transformer-based Multi-Camera 3D Object Detection

Q6: One very nearing lady is missed by the baseline PF-Track. I wonder if it is due to the difference in the detection part.

Thanks for your comment! In Table 7, S2-Track achieves comparable results with both detection heads, PETR (the default head of PF-Track) and DETR3D (the default head of S2-Track), indicating that the missed lady in the demo is not caused by the detection component. In fact, if you carefully review the video, you will notice that the nearby lady is detected in the first frame. However, due to interference from other vehicles and pedestrians, her bounding box is lost in subsequent frames. This is precisely where our carefully designed modules enhance performance, enabling stable and robust tracking in challenging scenarios.

Thanks again for your thoughtful feedback. We believe that your suggestions, along with our revision, have greatly enhanced the persuasiveness and completeness of our work. We hope our rebuttal can address your concerns.