STAR: Spatial-Temporal Tracklet Matching for Multi-Object Tracking

Dear Reviewer 5M6K,

Thank you for the detailed review and invaluable comments. We address your concerns point by point. Feel free to ask follow-up questions if something remains unclear.

---

Q1: It is suggested to resummarize its contribution to highlight its difference with existing tracklet graph matching-based methods works.

Respond to Q1: Thank you very much for the valuable suggestions. We have summarized the contribution of the STAR method again and clearly highlighted its differences from the existing tracklet graph matching-based methods. Firstly, unlike traditional graph-based MOT methods that consider each target independently, STAR considers not only the features of individual targets but also the interactions between targets (i.e., the topological relationships between targets). By establishing these relationships, STAR not only enhances the distinctiveness of individual target characteristics but also effectively addresses dynamic occlusion issues using the short-term stability in relationships between targets. This approach significantly improves tracking continuity and accuracy in complex environments.

Regarding the innovation in data association, STAR employs a graph matching approach where each target's tracklet segment is treated as a node in the graph. This method, compared to traditional strategies that consider each individual target as a node, greatly reduces the number of graph matching operations, thus significantly improving processing efficiency. In specific experimental tests, this technique not only drastically reduces the computational load but also enhances the smoothness and accuracy of tracking.

---

Q2: Some technique details: For object that haven’t appeared in a frame in the TC, they use an empty placeholder. How the appearance and position features and the topological features of such dummy nodes are computed? How to process them in the following graph matching.

Respond to Q2: For object that haven’t appeared in a frame in the TC, we use an empty placeholder. Then the initial features are initialized with the average of its adjacent non-empty frames.

After initialization, each tracklet segment is treated as a node, and the features of these nodes are iteratively updated through TCGC. Ultimately, $m$ feature graphs are obtained. By performing graph matching on these feature graphs, the matching between tracklet segments is achieved.

The ablation study analyzing the impact of STAR was conducted on the MOT16 dataset. We demonstrated that adding STAR to existing methods can further enhance their performance. Compared to Deepsort, JDE, and CTracker, CTracker*/Deepsort*/JDE* indicate that trajectories are generated using only the Intersection over Union (IoU) distance. The introduction of the STAR module consistently improves the performance of existing methods. The experiments indicate that STAR is a model-independent, universal method that can be seamlessly integrated into existing technologies, providing continuous performance optimization.

THank the authors for detailed rebuttal. My concerns about technique details are solved.

It is said STAR’s difference from existing methods lies in considering interactions between targets (i.e., the topological relationships between targets). In fact, modeling topological relationships for robust tracking has been considered in many existing MOT works, just listing 3 examples:

[24]Learning a Robust Topological Relationship for Online Multiobject Tracking in UAV Scenarios

[a]Probabilistic Tracklet Scoring and Inpainting for Multiple Object Tracking, CVPR2021

[b]Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking

The above works use neighborhood and topological relationships either spatially or both spatially and temporally. Some methods use the same features (i.e., relative distances and relative angles between the target and its neighbors ) as STAR to represent topological relationships, while some use other interaction or geometric features. Broadly speaking, the idea is widely accepted in the MOT domain that effective modeling of interactions will lead to better tracking quality as the motion of each target may be affected by the behaviour of its neighbors in the scene.

As another innovation in data association, STAR treats each target's tracklet segment as a node in the graph. This is also widely adopted in tracklet graph-matching based methods.

Dear Reviewer pdkw,

Thank you for the detailed review and invaluable comments. We address your concerns point by point and invite you to ask any follow-up questions if anything remains unclear.

---

Q1: The STAR aims to address the occlusion issues. However, the methods do not provide information or explanations indicating how each implementation contributes to the motivation to address the occurrence.

Response to Q1:

Thank you for your detailed review. With respect to the mechanism of STAR in managing occlusions, we have enhanced feature discernibility in occluded scenarios by integrating both temporal information and topological relationships into our system. Let me further clarify the application of these methods:

Temporal Information: By generating tracklets and extracting features over time, STAR enables the system to infer the characteristics of a object even when direct observation is momentarily obstructed. For instance, in a dense crowd, should an individual be temporarily hidden by others, the system continues to project their likely whereabouts until they become visible again.

Topological Relationships: The spatial interrelations among objects serve as another robust strategy to address occlusions. In scenarios like vehicle tracking, the relative positions of multiple objects typically exhibit short-term consistency. Leveraging this attribute, the STAR system preserves the integrity of the topological structure among objects, thereby facilitating the recognition and tracking of occluded objects.

To validate the efficacy of these approaches, rigorous testing was conducted using the dancetrack dataset, known for its frequent occlusions. The outcomes affirm that the fusion of temporal and topological information elevates the HOTA to 55.9%. This underscores the substantial improvement in handling occlusions, attesting to the robustness of our method.

---

Q2: In the literature review and benchmarks, some state-of-the-art methods are missing, such as BoosTrack++ and OccluTrack. Moreover, the performance on the MOT 17 and MOT 20 datasets is not comparable with some of those methods. Additionally, HOTA should be included in the benchmark, which is the most important metric for evaluating tracking performance.

Response to Q2: We acknowledge that our literature review did not include BoosTrack++ and OccluTrack. However, please understand that there are numerous studies related to MOT, and we have tried our best to keep our data fair and reliable .

To address these issues, we have included comparisons with these methods on the MOT 17 and MOT 20 datasets, and have introduced the HOTA evaluation metric. This will help in more comprehensively and accurately evaluating and demonstrating the performance of our model. It can be observed that our HOTA and MOTA scores are comparatively higher, while our IDF1 and ID Sw. scores are somewhat lower. Despite this, these outcomes clearly demonstrate the effectiveness of our method.

---

Q3: Provide the ablation study on the IoU-based Association.

Response to Q3:

We realize the significance of this component within our tracking framework. We have now conducted a detailed ablation study to evaluate the role of the IoU-based Association. DeepSORT*, JDE*, and CTracker* represent modified DeepSORT, JDE, and CTracker, which rely solely on IoU distance. Results from this study show that CITG perform best. We appreciate your suggestion and believe that this addition strengthens our paper significantly.

Thank you once again for your valuable feedback, and we look forward to your further queries or suggestions.

Dear Reviewer MVNd,

Thank you for the detailed review and invaluable comments. We address your concerns point by point and invite you to ask any follow-up questions if anything remains unclear.

---

Q1: Section 3.2 is quite brief and lacks details; for instance, how is the IoU threshold dynamically adjusted based upon the mentioned factors (object velocity, detection box size, and inter-frame time intervals)?

Response to Q1: A short-term trajectory $g_k= \{o_{1}^{k},o_{2}^{k},\cdots,o_{n_k}^{k}\}$ continues to be associated with the detection in the next frame until it no longer matches any detection. The Confident Initial Tracklet Generator utilizes IoU (Intersection over Union) to achieve the association of objects by calculating the overlapping extent between detection boxes. Here is the specific process:

1. Trajectory Initialization

   In the first frame of the video, each detection box is initialized as an independent short-term trajectory.

2. IOU Calculation

   For each trajectory in the current frame, the IoU value with all detection boxes in the next frame is calculated. The IoU value represents the overlapping extent between two detection boxes, ranging from $[0, 1]$, where a higher value indicates a greater degree of overlap.

3. Trajectory Matching

   Matching of trajectories and detection boxes is based on the IoU values: if the IoU value of the last detection box of a trajectory with a detection box in the next frame exceeds a certain threshold, they are considered to belong to the same object. If a detection box matches multiple trajectories, none are selected. Unmatched detection boxes are initialized as new trajectories.

4. Dynamic Adjustment of IOU Threshold

   To accommodate the motion characteristics of targets, a dynamic IoU threshold adjustment method based on multiple factors is adopted. Specifically, the IoU threshold is dynamically adjusted considering the target's motion speed, detection box size, and the time interval between frames.

   $\tau = \left( \tau_{0} - \alpha \cdot v + \beta \cdot \log(A) + \gamma \cdot \Delta t \right)$ $\tau = \max(\tau_{\text{min}}, \min(\tau_{\text{max}}, \tau))$

   Here, $v$ is the object's motion speed (calculated using the Euclidean distance between center points across frames), $A$ is the detection box area, and $\Delta t$ is the time interval between frames. The motion speed, detection box size, and time interval dynamically influence the IoU threshold.

5. Trajectory Update and Termination

   Matched detection boxes are added to corresponding trajectories, updating the trajectory's state (such as location and timestamp). Trajectories that remain unmatched for multiple consecutive frames (more than 3 frames) are considered terminated. Unmatched detection boxes initiate new trajectories.

6. Output Short-Term Trajectories

   These steps generate a set of short-term trajectories containing the preliminary motion trajectories of all targets in the video, serving as the foundation for subsequent trajectory association and long-term trajectory generation.

---

Q2: Has there been a study conducted to verify that the CITG makes "tracklet initialization more robust across diverse tracking scenarios" (L130)?

Response to Q2: Thank you for emphasizing the importance of the ablation study on the CITG. We realize the significance of this component within our tracking framework. We have now conducted a detailed ablation study to evaluate the role of the IoU-based Association. DeepSORT*, JDE*, and CTracker* represent modified DeepSORT, JDE, and CTracker, which rely solely on IoU distance. Results from this study show that CITG perform best. We appreciate your suggestion and believe that this addition strengthens our paper significantly.

This study underscores the robustness of CITG in diverse scenarios.

---

Q3: While the paper provides extensive quantitative results and ablation studies, it lacks a qualitative or case-specific analysis of where and why STAR may still fail (e.g., in highly dynamic or extremely long occlusion cases).

Response to Q3: STAR failure in two typical cases. The first case involves a small-scale pedestrian who is occluded for an extended period while being captured by a rapidly moving camera. In this scenario, the combination of the pedestrian's small size and the camera's fast motion can lead to significant deviations in the pedestrian's trajectory. To address this problem, it is recommended to utilize more advanced technologies, such as optical flow, which can provide improved motion compensation for the fast-moving camera.

The second failure case arises from false detections produced by the object detector. For instance, when the detector erroneously identifies a non-existent ID, the tracking algorithm incorrectly assigns a track to this false detection. To tackle this challenge, it is vital to develop more robust object detectors or to refine our approach into an integrated detection and tracking method, which can significantly reduce such errors.

---

We hope these explanations address your queries effectively. Thank you for your attention to our work, and we look forward to your further questions or suggestions.

Dear Reviewer pCN3,

Thank you for your detailed review and invaluable comments. We will continuously refine our expressions and abbreviations to further enhance readability. We address your concerns point by point and welcome any follow-up questions for clarifications.

---

Q1: If time permits, the authors could analyze whether there should be some degree of overlap or skipped-frame sampling for the frame sampling of Slicing TCs. This is because overlap can provide implicit temporal feature propagation, which can implicitly extend the length of the temporal sequence under consideration.

Response to Q1: We deeply appreciate the reviewer's scrutiny and suggestions, which are crucial for enhancing our work. Regarding your recommendations on overlapping and skipped-frame sampling strategies, we conducted an in-depth study to explore the best possible configurations.

Previously, we adopted a sampling strategy with a sliding window size of N=4 and a stride length of L=2. Based on your advice, we experimented with various combinations of window sizes and stride lengths to evaluate changes in model performance. The specific experimental results are as follows.

When the stride length is L=1, regardless of whether the window size is N=4 or N=6, the model performance was optimal. The smaller stride length, L=1, allowed us to more effectively capture continuity and changes over time, thus enhancing the model's ability to process temporal data.

Simultaneously, we compared the effects of stride lengths L=2 and L=1 with a window size of N=4 and found, although both configurations performed well, the L=1 setup allowed the model to maintain high accuracy while improving efficiency.

---

Q2: Do the authors have more data to explain the impact of the number of TCs (number of frames retained) and the number of frames retained in the expanded graph on experimental performance?

Response to Q2: Thank you for your detailed feedback. We understand your concerns about the tracking length. After evaluating performances for N=4, 5, 6, 7, 8, which is shown in the table below. Although there was a slight improvement at N=6, it was not significant. Therefore, we opted for N=4 to ensure robust performance in challenging scenes.

We recognize that N=4 may not seem to cover a long-range, but based on our research, this duration is already ahead of many existing studies where tracking usually does not exceed 4 frames. As such, even 4-frame tracking represents an advancement.

---

Q3: The paper unfolds the graph in a temporal sequence to achieve matching. However, how are cases handled where multiple TC (Tracklet Clip) matching results are inconsistent?

Response to Q3: Thank you for your question, it addresses a core aspect of our method. To handle frequent ID switches, our system sets strict criteria in the "Confident Initial Tracklet Generator" stage, where tracklet clips must meet rigorous conditions to be deemed "confident". Instances where IDs such as 1 and 2 switch frequently are not classified as confident, ensuring that unstable or low-confidence segments don't adversely influence the matching process. This approach enhances the robustness of our method. Our experiments on the Dancetrack dataset validate its effectiveness even amidst frequent ID switches.

---

Q4: The authors did not analyze how much impact the proposed module has on inference performance.

Response to Q4: Thank you for your valuable comments. We fully understand your concern about how the proposed module impacts inference performance. Indeed, the graph matching is a quadratic programming problem which could affect speed. To mitigate this, we introduced the Gated Search Tree algorithm in inference stage, which enhances efficiency through these three key steps:

1. Bipartite Graph Construction: We construct bipartite graphs between trajectory segments, establishing connections only under specific appearance and motion conditions.

2. Connected Components Identification: We use depth-first search to reduce the scale of matching computations.

3. Parallel Computation of Matching Costs: We parallelize the computation of matching costs across connected components, using a streamlined optimization approach for rapid processing.

The algorithm significantly accelerates inference speed through parallel processing while maintaining accuracy. We hope these explanations address your concerns, and we appreciate your thorough review of our research. We look forward to your further feedback.