Towards Category Unification of 3D Single Object Tracking on Point Clouds

Thanks for your valuable comment very much. We have polished our paper according to your suggestions. Here, we number and address questions as follows. Changes are highlighted in $\textcolor{red}{red}$ in the revised manuscript.

(1) Related works about idea. According to your insightful comment, we have discussed related works on our key idea to give a better understanding of our work. Our core idea is to encode adaptive group configurations to accommodate shape- and size-changed geometric information from cross-category data. We appreciate your observation regarding the connection between the mentioned references and our core idea, specifically the dynamic receptive field. We have added a subsection, as shown in the $**“2.3 Dynamic Grouping” subsection of the main manuscript on page 3, line 136**$.

(2) Latency comparison. According to your comment, we have conducted a comparison of latency between our unified models and eight state-of-the-art methods (including the methods you mentioned), aiming to give a comprehensive assessment of the proposed methods in comparison to existing methods. In addition, we have also included a comparison of model parameters and FLOPs. Our methods introduce slight overhead compared to the baseline methods. For a detailed comparison and analysis, please refer to the $**“B.4 Computational Cost Analysis” subsection of the Appendix on page 16, line 585**$.

(3) Recent methods comparison. To better demonstrate the effectiveness of the proposed category-unified models, we have compared the recent SOTA methods, such as TAT [ACCV2022], STNet [ECCV2022], DMT [T-ITS2023], CXTrack [CVPR2023] and SyncTrack [ICCV2023] for comparison, as shown in the $**“4.2 Comparison with State-of-the-art Methods” subsection of the main manuscript on page 7, line 295**$.

(4) Verification on Waymo. In the community, KITTI and NuScenes are often employed for a comprehensive evaluation of a tracker. Specifically, tracking models are typically trained and tested on each category within these two datasets. In contrast, Waymo is often utilized to assess the generalization capabilities of the models. A common practice is to test a model pretrained from KITTI on the Waymo dataset. To further validate the generalization of our unified model and enhance the quality of our work, we have evaluated the proposed category-unified model on Waymo, following common experiment setup. Detailed experiments and analyses are provided in the $**“B.3 Results on Waymo Open Dataset” subsection of the Appendix on page 16, line 571**$.

(5) AdaFormer for general detection. Thanks for your meaningful comment. We believe that the concept of AdaFormer holds potential for extension to general detection. To the best of our knowledge, 3D point cloud-based object detection can be broadly categorized into indoor and outdoor (autonomous driving scenarios) detection. From a model perspective, detection methods can be further classified into those relying on point representation and those utilizing voxel representation. Currently, in indoor scenes, methods based on point representation dominate, while in outdoor scenes, voxel representation-based methods prevail due to the computational challenges posed by large-scale scenarios for point representation-based methods. Our AdaFormer is a point representation-based feature extraction network. Its design philosophy may be extended to point representation-based detection methods, introducing dynamic receptive fields to enhance detection accuracy. For detection methods based on voxel representation, our approach may also offer insights, such as dynamically adjusting voxel sizes to enhance feature representation for objects with diverse shapes and sizes.

(6) Statistical analysis for supporting motivation. We can provide other statistical analysis to reveal diverse size of different categories to further support our motivation. Our goal is to design a category-unified model capable of tracking any object regardless of its category. The challenge we face is that different object categories exhibit distinct data distributions (size and shape) and learning objectives. Therefore, our motivation is to handle different object categories with diverse sizes and shapes in a unified manner for category-unified tracking. In the Appendix of the old manuscript, we have already visualized the differences in learning objectives as shown in Fig. 8 (in the new manuscript). To support our motivation from various perspectives, we have visualized the statistical analysis of size distributions, motion state distributions and background distraction distributions for the Car and Pedestrian categories on KITTI, and then performed a comparison analysis between the two categories. The presented statistical analysis can effectively support our motivation. Details can be found in the $**“B.1 Motivation Analysis” subsection of the Appendix on page 13, line 519**$.

Thanks very much for your positive feedback, we have polished our paper according to your valuable suggestions. We have incorporated a comparison of inference speed and FLOPs to enhance the quality of our work. We have also presented the model parameters. The proposed category-unified models introduce a tolerable additional inference cost and computational overhead compared to baseline methods (For example, P2B [48 Fps, 4.30 GFLOPs, 1.34M parameters] v.s. SiamCUT [36 Fps, 4.51 GFLOPs, 2.06M parameters]), mainly incurred by the proposed AdaFormer backbone. Our unified model inputs and learning objective do not incur any additional overhead during the testing phase. Detailed comparisons and analyses are provided in the **“B.4 Computational Cost Analysis” subsection of the Appendix on page 16, line 585, which is highlighted in \textcolor{red**{red} in the revised manuscript}.

Thanks for your valuable comment very much. We have polished our paper according to your suggestions. Here, we number and address questions as follows. Changes are highlighted in $\textcolor{red}{red}$ in the revised manuscript.

(1) Details of unified training. In our experimental configuration, the proposed unified model is trained on the full KITTI and then test on it, the same for NuScenes, not the combination of KITTI and NuScenes. More specifically, the experimental setup commonly used in the community is to train and test each category individually within each dataset. To be a fair comparison, our unified model is trained on all training data (cross-category data) of a dataset, and then tested on all test data (diverse categories) of that dataset. In addition, during the training phase, we directly use data from all categories to train a unified model, without any additional data augmentation techniques. We appreciate the insightful comment and have made improvements to the manuscript by supplementing training details, as shown in the $**“A More Implementation Details” section of the Appendix on page 13, line 497**$.

Further discussion: There are notable differences among various datasets. For example, NuScenes displays more pronounced sparsity compared to KITTI and Waymo. We posit that cross-dataset training requires addressing challenges related to data distribution. This may include dealing with diverse background and motion state distributions, such as differences in pedestrian behavior between KITTI and NuScenes. The concept of joint cross-dataset and cross-category training will enable a single model to track all objects across all categories, irrespective of data distribution, which further enhances the model's generalization capabilities. In the future, we will keep it tune and further improve our unified model.

(2) Evaluation on Waymo. According to your comment, we have evaluated the proposed method on Waymo. Unlike KITTI and NuScenes, the community often conducts generalization experiments on Waymo dataset. Following this, we test the KITTI pre-trained model on Waymo. We also compare our category-unified models with existing SOTA methods, including the recent methods you mentioned. The experimental results demonstrate that the proposed category-unified model not only generalizes well across diverse categories but also exhibits a certain level of generalization to datasets with varied data distributions. Detailed experiments and analyses are provided in the $**“B.3 Results on Waymo Open Dataset” subsection of the Appendix on page 16, line 571**$.

(3) Recent methods comparison. To better verify the effectiveness of the proposed approach, we have included all the recent methods you mentioned and the latest SyncTrack [ICCV2023] for comparison, and added the corresponding comparison analysis, as shown in the $**“4.2 Comparison with State-of-the-art Methods” subsection of the main manuscript on page 7, line 295**$. In addition, we also compare our category-unified models with recent SOTA methods on Waymo, as shown in the $**“B.3 Results on Waymo Open Dataset” subsection of the Appendix on page 16, line 571**$.