CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features

Dear Reviewer f47k,

Thanks for your time and effort in reviewing our work. Your recognition of our novel method, comprehensive experiments, SOTA performance, and writing quality greatly encourages us. We hope the following responses can address your concerns.

---

Q1: Comparison with recent works

Thank you for mentioning some recent trackers. Here’s how our CSTrack differs:

1. TATrack and MCTrack use two symmetric backbones for separate RGB and X processing. ViPT employs an asymmetric X branch, while USTrack and Yang [1] concatenate RGB and X feature tokens for multimodal feature integration with a one-stream backbone. However, they still need to process RGB-X dual inputs within the backbone. M3PT uses middle fusion but relies heavily on the early dual-branch architecture for performance.
2. In contrast, our CSTrack integrates RGB-X dual inputs into a compact feature space, reducing model complexity and computational costs. Additionally, the table below compares various trackers, highlighting the effectiveness of our method. |Tracker|Lasher (SR, PR)|RGBT234 (MSR, MPR)| |-|-|-| |TATrack|56.1; 70.2|64.4; 87.2| |MCTrack|57.1; 71.6|65.6; 87.5| |ViPT|52.5; 65.1|61.7; 83.5| |USTrack|- |65.8; 87.4| |Yang [1]|56.7; 71.4|65.1; 87.3| |M3PT|56.1; 70.0|63.9; 86.5| |CSTrack-B|60.8; 75.6|70.9; 94.0|

---

Q2: Further explanation of model details

1. Basic model setup. CSTrack is a unified tracker trained jointly across different datasets, enabling testing without modal differentiation. This is mentioned in Section 4.1 and will be included in the method section of the revised version.

2. SCM description. We apologize for the error in Equation 3 of our paper. The RGB related features should be processed as $[q'_r; f^{'t}_r]$, not $[q_r; f^{t}_r]$. Below is the corrected equation, confirming our use of the serial computation method in Figure 1.

   $[q' _x; f^{'t} _x] = Norm( [q _x; f^{t} _x] + \Phi _{CA} ([q _x; f^{t} _x], [q' _r; f^{'t} _r]))$

---

Q3: Further explanation of experimental results

1. Ablation study of core modules. The table below presents ablation results for key modules: Spatial Compact Module (SCM) and Temporal Compact Module (TCM). Adding SCM and TCM improves performance, validating our design. We also offer Lasher benchmark results, consistent with performance rankings observed in RGBT234. |Setting|DepthTrack (F; Re; PR)|RGBT234 (MSR; MPR)|Lasher (SR; PR)|VisEvent (SR; PR)| |-|-|-|-|-| |Baseline|63.6; 64.1; 63.1|68.2; 92.4|57.6; 72.9|62.9; 79.6| |+ SCM|65.1; 65.3; 64.9|69.6; 93.0|59.8; 74.7|64.1; 81.0| |+ TCM|65.8; 66.4; 65.2|70.9; 94.0|60.8; 75.6|65.2; 82.4|

2. SCM analysis. Table 6 shows cross-attention and shared embedding in SCM boosts DepthTrack but impacts RGBT234 and VisEvent less. We attribute this to higher noise and fluctuations in depth data compared to thermal and event data, as detailed in Figure 3 and our supplementary videos. This suggests these designs are most effective in challenging scenarios. For RGBT234 and VisEvent, other designs within SCM are sufficient for strong performance, as shown by SCM's gains across all benchmarks (see the table above).

3. Tracking speed. In Table 5, our method achieves 35 FPS, surpassing dual-branch trackers at 24 and 18 FPS. This meets expectations and satisfies real-time tracking requirements (exceeding the VOT challenge threshold of 20 FPS).

---

Q4: Additional experimental analysis

1. Performance with ViT backbone. We retrain CSTrack using the ViT backbone. The results below demonstrate our compact modeling exceeds the performance gains of the HiViT backbone, and our tracker achieves SOTA performance even with the ViT backbone, validating our approach's effectiveness. |Backbone|Compact Modeling|DepthTrack (F; Re; PR)|Lasher (SR; PR)|VisEvent (SR; PR)| |-|-|-|-|-| |HiViT| ✔ |65.8; 66.4; 65.2|60.8; 75.6|65.2; 82.4| |HiViT| ✘ |63.6; 64.1; 63.1|57.6; 72.9|62.9; 79.6| |ViT| ✔ |63.9; 64.8; 64.3|58.6; 74.3|64.6; 81.1|

2. RGB-only tracking test. Our core compact modeling targets RGB-X dual-stream inputs, making RGB-only tracking not aligned with our objectives. Therefore, we did not conduct this test.

3. Impact of bidirectional cross-attention order. We fine-tune our model by swapping the cross-attention order, and the results show minor performance fluctuations. We speculate that the shared patch embedding and the subsequent one-stream backbone can diminish the influence of interaction order. |Setting|DepthTrack (F; Re; PR)|Lasher (SR; PR)|VisEvent (SR; PR)| |-|-|-|-| |CSTrack-B|65.8; 66.4; 65.2|60.8; 75.6|65.2; 82.4| |+ swap order|65.8; 66.0; 65.4|60.7; 75.8|65.2; 82.4|

---

We hope these responses address your concerns and kindly invite you to reassess your rating. Feel free to reach out to us if you have any further questions.

---

[1] From Two Stream to One Stream: Efficient RGB-T Tracking via Mutual Prompt Learning and Knowledge Distillation, Yang et al., in arxiv 2024.

Dear Reviewer W7oh,

We sincerely appreciate your thorough review of our work and are grateful for your recognition of our sound motivation, effective method, and extensive experimental analysis. In response to your concerns, we have provided detailed explanations below:

---

Q1: Noise Interference in modality-missing tracking

Thanks for your constructive suggestion on analyzing noise impact in modality-missing tracking task. IPL [1] recently proposes the modality-missing tracking benchmarks, along with a baseline method that duplicates available modality data, which we adopt. As shown in Tables 2 and 4 of the paper, modality-missing settings significantly degrade model performance. We attribute this mainly to the influence of two types of noise:

1. Miss of advantage modality. Figure 4 in the paper introduces the concept of the advantage modality, which primarily conveys object appearance information in challenging scenarios. In modality-missing task, this advantage modality may be unavailable, leaving the tracker susceptible to noise interference from the non-advantage modality due to its lack of appearance features.

2. Training and inference bias. CSTrack is trained with complete RGB-X modalities but directly inferred in modality-missing settings. The difference in input data types introduces noise that significantly affects performance. Despite this, CSTrack exceeds IPL, a tracker tailored for such scenarios. We think this is primarily due to our shared patch embedding, which transforms initial RGB and X data into a unified feature space, improving compatibility with varied inputs. Below, we test the impact of shared embedding to further confirm our analysis: |Setting | LasHeR-Miss (SR; PR) | RGBT234-Miss (MSR; MPR) | |-|-|-| |Unshared Embedding | 48.7, 60.6 | 65.3, 86.5 | |Shared Embedding | 50.3, 62.2 | 68.0, 89.8 |

---

Q2: Distribution analysis of RGB and RGB-X datasets

We appreciate your valuable suggestions on the distribution analysis of various modality datasets. The analysis is as follows:

1. Distribution differences across modality datasets. We perform statistical analyses on representative RGB and RGB-D/T/E datasets to obtain normalization mean values of training videos, including the ImageNet for comparison as a reference for natural image distribution: |Datasets| ImageNet | LaSOT (RGB-only) |DepthTrack (RGB-D)|Lasher (RGB-T)|VisEvent (RGB-E)| |:-|:-|:-|:-|:-|:-| | RGB Mean| 0.485; 0.456; 0.406 | 0.456; 0.459; 0.426 | 0.417; 0.414; 0.393 | 0.500; 0.499; 0.471 | 0.418; 0.375; 0.317 | | X Mean| - | -| 0.574; 0.456; 0.240 | 0.372; 0.372; 0.368 | 0.935; 0.904; 0.949 |

   • RGB modality: LaSOT and DepthTrack, collected in typical natural environments, have mean values similar to ImageNet. However, Lasher, often focusing on dark environments, and VisEvent, which collects predominantly yellow RGB images, deviate from these natural image means.

   • X modality: as shown in Figure 3 of the paper, X modalities vary greatly in image style, resulting in significant differences in mean values.

2. Model's discriminative ability across modality datasets. Using features from the one-stream backbone (see Equation 8 in the paper), we extract tokens associated with learnable queries and build a four-class classifier with two fully connected layers to identify the input modality type: RGB-only, RGB-D, RGB-T, and RGB-E. We freeze the existing model and train this classification head for 2 epochs. The accuracy results show the model can effectively distinguish different modality datasets, thanks to the clear distribution differences. |Datasets|LaSOT (RGB-only)|DepthTrack (RGB-D)|Lasher (RGB-T)|VisEvent (RGB-E)| |:-|:-|:-|:-|:-| |Accuracy|100%|98%|100%|100%|

3. Advantages of joint training. Our ablation study, detailed in Table 8 (#3) of the paper, shows that the benefits from joint training, including larger datasets and knowledge sharing across modalities, outweigh the drawbacks of domain shift, resulting in enhanced model performance.

---

Q3: Function of bidirectional cross-attention and modality-specific queries.

Our spatial compact module aims to integrate RGB and X input streams into a unified feature space, facilitating simplified and effective spatial modeling. When a modality has low-quality or noisy data, termed "non-advantage modality" (see Figure 4), bidirectional cross-attention first emphasizes the advantage modality's representations while minimizing those of the non-advantage modality. Next, modality-specific queries preserve global semantic information of each modality, offering additional reference information for subsequent feature integration.

---

We hope these explanations can resolve your concerns. If you have any more questions or need further clarification, feel free to reach out to us.

---

[1] Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarks, Lu et al., in IJCV 2O24.

Dear Reviewer waVm,

Thank you for your time and effort in reviewing our work. We appreciate your recognition of our novel, innovative, and efficient method, along with the comprehensive, well-structured, and convincing experiments, as well as easy-to-follow writing and the extensive supplementary material. We note your concerns about the performance and computational efficiency of our model. In response, we've conducted targeted comparative analyses and additional experiments, aimed at addressing your concerns.

---

Q1: Performance comparison with STTrack [1]

We provide a detailed comparison between CSTrack and STTrack across various metrics:

1. CSTrack-B, the model presented in our paper, exhibits superior average performance. As noted, CSTrack-B exhibits performance comparable to STTrack in VOT-RGBD22 (EAO ↓ 0.2%; Acc ↑ 0.8%; Rob ↓ 0.8%) and Lasher (SR ↑ 0.5%; PR ↓ 0.4%). However, CSTrack-B significantly outperforms STTrack in other benchmarks, achieving improvements such as PR ↑ 2.0% in DepthTrack, MPR ↑ 4.2% in RGBT234, and PR ↑ 3.8% in VisEvent, contributing to a superior average performance.

2. CSTrack targets a more challenging optimization objective. Unlike our CSTrack, which uses a single model weight to simultaneously handle RGB-D/T/E tasks, STTrack optimizes separate weights for each task. Additionally, the STTrack repository suggests that it even optimizes individual checkpoints for each benchmark. While UnTrack [2] indicates that this approach may yield better performance, it is a less practical strategy for real-world applications.

3. CSTrack-B offers superior computational efficiency. CSTrack-B offers a substantial advantage in Params (↓ 53M) and Flops (↓ 55G), achieving a better performance-computation balance. Focusing on maximizing performance, we develop CSTrack-L by adopting a larger backbone, surpassing STTrack across all benchmarks.

---

Q2: FLOPs reduction compared to dual-branch methods

As shown in Table 5 of the paper, our single-branch method reduces FLOPs by only 2G compared to asymmetrical dual-branch methods. Here's the explanation:

1. Asymmetrical dual-branch methods often incorporate parameter-intensive prompter modules into each layer of the RGB branch to integrate RGB-X features, as seen in the fully connected networks in OneTracker [4]. This results in some FLOPs benefits but leads to increased parameters (↑ 9M) and speed limitations (↓ 11 FPS).

2. Table 5 indicates a notable disadvantage in tracking performance with this dual-branch method, e.g., MSR ↓ 3.7% in RGBT234. It suggests that the scale of FLOPs in this method makes it difficult to effectively integrate RGB-X features.

---

Q3: Computational efficiency comparison with existing methods

Thanks for your constructive suggestion. We analyze the computational efficiency of recent open-source RGB-X methods, yielding the following results:

CSTrack-B excels in Params and Speed, with only Flops lagging behind UnTrack. Similar to the analysis in Q2, UnTrack's core modules primarily consist of fully connected networks, offering Flops advantages but limits Params, Speed, and tracking performance. Overall, our method demonstrates superior tracking performance and computational efficiency.

---

We hope these responses address your concerns and would be grateful if you could reconsider your rating. Should you have any additional feedback or require further clarification, please do not hesitate to let us know.

---

[1] Exploiting Multimodal Spatial-Temporal Patterns for Video Object Tracking, Hu et al., in AAAI 2025.

[2] Single-Model and Any-Modality for Video Object Tracking, Wu et al., in CVPR 2024.

[3] SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking, Hou et al., in CVPR 2024.

[4] OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning, Hong et al., in CVPR 2024.