Revisiting motion information for RGB-Event tracking with MOT philosophy

We want to express our gratitude for your support and valuable insights. It's reassuring to know that you recognize the originality and efficacy of our research. We would greatly appreciate your continued support in championing our work.

W1 and Q6. More experimental results under varying illumination conditions

Advantages and disadvantages of event stream: As discussed with FE108, VisEvent and COESOT, event cameras (Dynamic Vision Sensors (DVS)) excel at capturing motion information at very low latency, and are almost free from the trouble of motion blur. In addition, DVS sensors also outperform the visible (RGB) cameras on dynamic range (140 vs 60dB), which enables them to work effectively even under poor illumination conditions. Simultaneously, we noticed that the event camera cannot provide effective information for tracking of slow-moving or stationary objects.

Input representation: Previous event trackers usually transform the asynchronous event flows into synchronous event images by stacking the events in a fixed time interval. CEUTrack transforms the event points into voxels which preserves the temporal information well, and the voxels top-selection mechanism also fully exploits the sparse representation of event data which reduces the computational complexity. In this paper, we also adopt the same transformation to get the event images but focus on designing a novel tracking framework to explore both the effective appearance information and motion information. Besides, the proposed CSAM framework employs both the appearance information and the motion information to suppress the noisy information within the event data.

Cannot be directly applied to other multimodal tracking tasks: Considering the motion information provided by the event data, in the proposed STTE, we design a spatial transformer encoder to establish distinctive spatial relationships by using an event stream. In addition, in the proposed DBTD, a motion decoder is introduced to match each tracklet by using the event tokens. Differently, for other multi-modal tracking tasks (e.g., RGB-Thermal, RGB-Depth), it is still difficult for us to construct accurate spatial correlations or match tracklets by using other modality data like thermal images or depth images.

Attributes Performance: Here, in order to prove the necessity of DVS, we provide the results of with and without event stream under the challenges of low illumination (LI) and Illumination variation (IV). As shown below, our tracker with event stream shows substantially better results, which demonstrates the effectiveness of DVS.

W2. Too many abbreviations.

we are very sorry for the confusion. In the revision, we will avoid unnecessary abbreviations, and the major mathematical symbols used in this paper will be summarized in a Table for ease of explanation in the revision.

Q1. Latency of the CSAM method.

Latency of NMS: In this Table, we provide the time each module takes. We found that NMS takes very little time in the PyTorch framework. This is because the tracking scenes usually contain only a limited number of candidates. Besides, we can use the cuda version of NMS to speed up its operation.

Latency of Hungarian algorithm: Similarly, due to the limited number of candidates, the Hungarian algorithm used in multi-target tracking will not take too long. To model the temporal associations in historical frames, the temporal encoder occupies a longer reasoning time in the CSAM framework. In the future, we will study how to use a more compact representation of historical information to efficiently establish temporal associations.

Anchor-free: Our appearance model follows the anchor-free pipeline to generate a series of candidates. The candidate matching network also does not use anchor boxes. We argue that the design of a one-stage framework in the future can improve the tracking efficiency; this will be our follow-on work.

Q2. The FPS and FLOPs of SeqTrackv2-B256.

SeqTrack casts visual tracking as a sequence generation task, forecasting object bounding boxes in an autoregressive manner. Such a design requires the gradual generation of tracking results and frequent access to intermediate calculation results, resulting in lower FLOPs and slower inference speed.

Q3. More examples of appearance and disappearance of targets.

We will add more examples in the revision. We also provide the tracking performance under the challenges of FOC (target is fully occluded) and OV (target completely leaves the video sequence), we noticed that our framework significantly improves the tracking performance when targets frequently appear and disappear.

Q4. The total training time.

Our proposed model is trained in two stages. In the first stage, the appearance model’s training took 12 hours. In the second stage, the proposed model took 7 hours for training.

Q5. Different metrics in FE108, COESOT and VisEvent

As in FENet[38], to mitigate small alignment errors, we utilize two widely used metrics, i.e., maximum precision rate (MPR) and maximum success rate (MSR), to evaluate the tracking performance on FE108. Differently, VisEvent and COESOT employ better alignment, they directly use the PR and SR metrics to evaluate different trackers. All evaluations are tested under the official codes.

We greatly appreciate your constructive and insightful comments. Thanks for your recognition of our work.

Q1. Putting some qualitative analysis in the main text.

Due to the limited page space, our previous manuscript only provides the qualitative analysis in the supplementary materials. We will put some of them in the main text in the revision space permitting.

In addition, please allow me to restate the novelty of our work briefly:

1. To the best of our knowledge, we are the first to introduce the MOT philosophy for the SOT task using RGB-E data. The proposed CSAM framework significantly improves the tracker's ability to cope with tracking drift caused by distractors.

2. We propose three effective modules: a Candidate Encoding Module, a Spatial-Temporal Transformer Encoder and a Dual-branch Transformer Decoder. The Candidate Encoding Module initially aggregates the RGB features and corresponding classification scores to generate the appearance embedding. Meanwhile, the event features and corresponding bounding box coordinates are fused to obtain motion information. In the Spatial-Temporal Transformer Encoder, the spatial encoder construct spatial correlations of each frame by using motion information, and the temporal encoder establishes the temporal relationships of each tracklet. In the proposed Dual-branch Transformer Decoder, we also generate the assignment matrix by using both the appearance information and motion information, simultaneously.

3. We show significantly improved state-of-the-art results of our proposed method on multiple RGB-E tracking benchmarks.

We extend our heartfelt gratitude for your perceptive insights and your recognition of the unique and meaningful contributions made by our research. Your support is highly valued, and we would be honored if you could serve as an advocate for our work.

Q1. The ablation study about N/M/T/$\tau_{th}$/$\zeta$:

N and M: N and M denote the number of candidates of the previous and current frame respectively. Therefore, N and M are actually determined by the tracking sequences and are not one of our settings. In supplementary C.5, we provide the influence of candidates to illustrate the effectiveness of the proposed framework.

The ablation study of T: The ablation study about T is shown below. When T=1, there is no temporal information included in the proposed framework. T > 1 can improve the association accuracy. In addition, since increasing T also adds more tracklets for the association, it increases the complexity of the association task. Finally, T = 15 is used in all experiments.

The ablation study of $\tau_{th}$: We show the ablation studies of $\tau_{th}$ below. As shown in the following table, we achieve the best performance when $\tau_{th}$=0.75. When $\tau_{th}$ is too small, the proposed tracker may cause false tracklet-matching. When $\tau_{th}$ is too big, the proposed tracker may not be able to complete the tracklet-matching.

The ablation study of $\zeta$: We show the ablation studies of $\zeta$ below. As shown in the following table, we achieve the best performance when $\zeta$=0.25 When threshold $\zeta$ is too big, it is hard to relocate the target.

Q2. The delay of the Hungarian algorithm.

Hungarian algorithm: As shown in the table below, the Hungarian algorithm takes 0.6 ms for object association. Because when we tested on the SOT dataset, most tracking scenes only contained a limited number of candidates (Please refer to Appendix C.5).

Other parts: To model the temporal associations in historical frames, the temporal encoder takes a longer reasoning time in the CSAM framework. In the future, we will study how to use a more compact representation of historical information to efficiently establish temporal associations.

Q3. Training process.

We do not use the sequential iterative manner during training. We employ a similar training strategy as that in KeepTrack, which has been carefully introduced in Appendix A and B. For partial supervision, we use two consecutive frames as the historical frames and select one frame to output the prediction results and calculate the training loss. For self-supervision training, we only require a single frame and its candidates for training loss calculation. We will add relevant content in the main text in the revision.

Q4. More layers in STTE and DBTD

We provide the experiments about using more layers in STTE and DBTD in the following Table. We found that a layer number of 2 can achieve better performance but at the expense of operational efficiency. Furthermore, more layers will introduce a large number of parameters, which may cause overfitting problems and lead to performance degradation.

Q5. More thorough checks are needed.

We are very sorry for the above mistakes in the previous manuscript. We will modify all of these typos in the revision.

We greatly appreciate your constructive and insightful comments. Thanks for your recognition of our work.

Q1. The motivation of the paper is unclear. Lack of analysis.

Motivation: As shown in Fig. 1 (a) in the manuscript, the co-occurrence of distractor objects similar in appearance to the target is a common problem in the SOT task. Most SOT trackers based on appearance information struggle to identify the target in such cases, often leading to tracking failure.

Existing RGB-E trackers primarily concentrate on exploring complementary appearance information within RGB and event data to enhance tracking performance. Despite achieving commendable improvements, mainstream RGB-E tracking algorithms still cannot solve the association problem of the targets and distractor objects in the temporal domain.

Some RGB trackers (e.g., KYS, KeepTrack, TrDiMP and DMTrack) propagate valuable scene information to improve the discriminative ability. However, these methods are susceptible to environmental interference, and their matching strategies relying on appearance information may miss the target when the target and distractor trajectories are close.

In fact, event data can not only provide the edge information to improve the RGB feature representations but also contains abundant motion cues to reflect the motion state of the objects, which is meaningful to differentiate between targets and distractors, even if they may look similar. The event camera offers a new perspective to reflect the motion state of the objects, which is crucial to accurately match the tracklet in the tracking sequences.

Therefore, we propose an Appearance-Motion Modeling RGB-E tracking framework to dynamically incorporate motion information as well as appearance information contained in the RGB-E videos to track all the candidates with a Multi-Object Tracking (MOT) philosophy, thus avoiding the tracking drift.

Experimental results: We provide the qualitative analysis (trajectory visualization) in lines 634-644 (Fig 9， Fig 10). In addition, we provide the visualization of candidate features via t-SNE in lines 629-633 (Fig 8). Furthermore, more ablation studies are covered in Appendix C to verify the effectiveness of the proposed framework. We will add more visualization results in the revision.

Q2. Spatio-temporal encoder-decoder is explored in SOT and MOT community

Different Motivations with existing SOT methods: Some SOT works (e.g., TrDiMP, TCTrack, HIPTrack) employ the encoder-decoder structure to explore historical information. Most of these methods do not explicitly distinguish the motion trajectories of distractors in the scene. The encoder and decoder are usually used to combine historical features and enhance current frame features, respectively. Our work is different, in the proposed CSAM framework, the encoder establishes the association between different candidates in each frame and the temporal relationships of each candidate. And the decoder is used to match candidates with historical tracklets.

Different Design with existing MOT methods: Numerous MOT methods use the encoder-decoder structure to construct the temporal relationships and generate the assignment matrix for matching. However, most of them rely on the appearance information from the input RGB data. These methods are susceptible to environmental interference, and their matching strategies relying on appearance information may miss the target when the target and distractor trajectories are close. Differently, in the proposed CSAM framework, taking into account the fact that event data can provide reliable motion status, we fully exploit the appearance and motion information in the tracking scene during the encoding and decoding stages to enhance tracking performance.

Q3. Improving Writing.

Sorry for the unclear description. We will modify the abstract in the revision as follows: RGB-Event single object tracking (SOT) aims to leverage the merits of RGB and event data to achieve higher performance. However, existing frameworks focus on exploring complementary appearance information within multi-modal data, and struggle to address the association problem of targets and distractors in the temporal domain using motion information from the event stream. In this paper, we introduce the Multi-Object Tracking (MOT) philosophy into RGB-E SOT to keep track of targets as well as distractors by using both RGB and event data.

Q4. The model is complex.

In lines 318-324, Table 2 shows that our CSAM introduces limited computation costs (5.6 ms latency and 14.4M parameters), while significantly improving the tracking performance. In addition, compared with recently advanced trackers, i.e., ViPT and SeqTrackv2-L256, our model size has not increased significantly and we can still run at real-time speeds.

Q5: No open source code.

We describe the proposed methods in Section 3 and provide the implementation details in Section 4.1, Appendices A and B. We illustrate the exact command and environment needed to run to reproduce the results. We will release the source code when the paper is accepted.