LRR: Language-Driven Resamplable Continuous Representation against Adversarial Tracking Attacks

Thank you for the comments. It appears that there might be some misunderstanding regarding certain details in the submission. The language utilized in Sec 3.3 is not predetermined; rather, it is extracted from the provided template using the CLIP model. Specifically, we initiate the process by employing CLIP's image encoder to extract the template's embedding. Subsequently, with a set of texts encompassing potential categories of the object, we compare the template's embedding with the embeddings of all the texts. Following this, we select the text embedding that exhibits the highest similarity with the template's embedding as the $\mathbf{z}_\text{txt}$ used in Eq. (6). Note that the text set can be updated based on different application scenarios. Alternative vision-language models or image caption methods can also achieve the same objective.

To address the concern, we have added an experiment by using the BLIP to extract the text embedding $\mathbf{z}_\text{txt}$. As results are shown in the following table, we observe that the two methods achieve similar results.

Table R-1: Using CLIP and BLIP to extract the text embeddings.

To avoid the misunderstanding, we have revised section 3.3 by adding the above discussions in the manuscript to highlight the details.

Thank you for the comments. The reviewer might neglect the results in Table 5 and the corresponding discussions. Here, we would like to clarify further the reasons for using SiamRPN++ as the main experimental setup and report the results on a state-of-the-art transformer method, i.e., ToMP-50 [Mayer et al. 2022].

Specifically, the main objective of this submission is to explore an effective way against SOTA adversarial tracking attacks. We select SiamRPN++ trackers [Li et al.,2019] as the main study subjects due to the following reason: all of the representative attacks including IoUAttack, SPARK, CSA, and RTAA have official codes or models to attack SiamRPN++ trackers. By conducting experiments on SiamRPN++ trackers, we can avoid the performance reduction issues caused by the re-implementation of different attacks and compare defense methods effectively.

To address the concerns, we highlight experiments on a transformer-based tracker in Table 5, i.e., ToMP-50 [Mayer et al. 2022]. Specifically, we use the RTAA to attack the ToMP-50 and use our method LRR for defense, and evaluate the results on three datasets. As shown in the following table, we see that: Firstly, RTAA can also lead to obvious accuracy reductions for the transformer-based tracker on three different datasets. Secondly, our method still presents powerful defense capability and keep high or even higher tracking accuracy than the results on clean data.

To address the concerns more comprehensively, our intention was to include the mentioned trackers in our evaluation. However, due to time constraints, we were only able to obtain results for OSTracker using its official code, as presented in Table R-4. It is worth noting that adapting existing attacks to new trackers would require a significant amount of time.

Given that ToMP-50 and the mentioned trackers are papers from the same time period, we believe the conclusions drawn demonstrate the effectiveness of our method on transformer-based trackers. We will involve the results of these trackers in our future revision. Here, we have added a discussion about this in Sec 2.

Table 5: Defense results on ToMP-50 across three datasets.

Table R-4: Defense results on OSTracker across one dataset.

[Mayer et al. 2022] C. Mayer, M. Danelljan, G. Bhat, M. Paul, D. P. Paudel, F. Yu, L. Van Gool. Transforming model prediction for tracking. in CVPR 2022.

Thank you for the suggestions. In the original supplementary material, we have reported the results on other three larger datasets, i.e., LaSOT, NFS30, and NFS240, in Appendix A.1 with Table 7.

Following the provided suggestions, we have conducted additional experiments on the TrackingNet and TNL2K datasets to validate the effectiveness of our approach.

For the TrackingNet dataset, we initiated four attacks (RTAA, CSA, IoUAttack, and SPARK) against the SiamRPN++ Res50. Subsequently, we employed our LRR to defend against these attacks. As illustrated in Table R-2, the four attacks substantially diminished the tracker's accuracy, and LRR demonstrated its capability to enhance the tracker's robustness under diverse attack scenarios.

Table R-2: SiamRPN++ ResNet50 Precision (%) on TrackingNet under four attacks.

We have added the results to Table 7 in Appendix A.1 and added explanations in Sec 4.2.

In relation to the TNL2K dataset, it comprises 700 sequences for testing, with 100 sequences specifically featuring adversarial examples generated using the RTAA. However, upon executing the SiamRPN++ on the provided 100 adversarial sequences, we observed a limited impact on the tracking accuracy. To mitigate any potential risks of misinterpretation, we conducted attacks on the remaining 600 sequences using the three attacks (i.e., RTAA, CSA, and SPARK) for a fair comparison, followed by employing our LRR for defense. Please note that IoUAttack is excessively time-consuming and cannot be executed within the given constraints. We plan to include it once the initial set of experiments is completed. As indicated in the results presented in Table R-3, our observations align closely with the findings from the TrackingNet experiments.

Table R-3: SiamRPN++ ResNet50 Precision (%) on TrackingNet under three attacks.

Thank you for the suggestions. It's important to note that natural language-based trackers have distinct designs and pipelines compared to template-based tracking methods. Despite this distinction, all existing attacks are tailored for traditional template-based trackers. We have observed that transferring these existing attacks to natural language-based trackers is not straightforward. We intend to delve into this challenge as part of our future work and to address this, we have incorporated a discussion in the conclusion section.

[Wang et al. 2021] X. Wang, X. Shu, Z. Zhang, B. Jiang, Y. Wang, Y. Tian, & F. Wu (2021). Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. in CVPR, 2021.

Thanks for the suggestions. We've diligently proofread the entire submission, rectifying some typos and grammar errors.

Thanks for the comments. We follow the released codes from existing tracking adversarial attacks (i.e., RTAA(Jia et al., 2020), IoU(Jia et al., 2021), CSA(Yan et al., 2020), and SPARK (Guo et al., 2020)) to implement attacks in our experiments. We detail some setups as follows. We also add a section in the supplementary material (i.e., Appendix A.9) with the explanations in Sec 4.

For RTAA, we utilized their originally released code (https://github.com/VISION-SJTU/RTAA/blob/main/DaSiamRPN/code/run_attack.py). The process follows these steps: 1. RTAA receives an incoming image and the target location where the image is the search region cropped by the studied tracker. 2. RTAA adds adversarial perturbations to the search region and outputs an adversarial example for the tracker to handle. At each frame, the attack optimizes the adversarial perturbation iteratively ten times, with the maximum perturbation set to 10/255. 3. RTAA outputs the optimized adversarial example as the new search region.

For the IoU Attack, we adhered to their default setups in their released code for conducting our experiments (https://github.com/VISION-SJTU/IoUattack/blob/main/pysot/tools/test_IoU_attack.py). Specifically, we follow the subsequent steps: 1. IoUAttack receives the frame and bounding box as inputs. 2. IoUAttack optimizes the perturbations iteratively until the IoU score is below the predefined score (See the released code for details). 3. IoU outputs the optimized adversarial frame to attack the tracker.

For CSA, we employed their released pre-trained perturbation generator to attack each frame (https://github.com/MasterBin-IIAU/CSA/blob/efd69a5705dd21c6701fd4ae7922f3a44647069a/pysot/pysot/tracker/siamrpn_tracker.py#L220). Specifically, CSA receives the clean search region and feeds it to the pre-trained perturbation generator. Then, the generator outputs the adversarial perturbation that is added to the clean search region.

In the case of SPARK (https://github.com/tsingqguo/AttackTracker/blob/main/tools/attack_oim.py), we employed the targeted attack approach provided in SPARK's default setup from their released code for attacks. The procedure involves the following steps: 1. SPARK takes the search region, cropped from the input frame, the targeted trajectory, and the targeted tracker as inputs. 2. SPARK conducts iterative optimization on the perturbations, iterating 10 times every 30 frames and 2 times at other frames. The maximum perturbation allowed is 0.3. 3. SPARK generates the optimized adversarial search region to attack the tracker.

Thanks for the suggestion. We implemented a resizing-based defense using the cv.resize operation in OpenCV. Specifically, for an input image $\mathbf{I}\in R^{H\times W}$, we first downsample it by a factor $r$ and get image $\mathbf{I}_\downarrow\in R^{rH\times rW}$. Then, we upsample it to the raw resolution, generating the reconstructed image $\hat{\mathbf{I}} \in R^{H\times W}$. Following this, we input the reconstructed images into trackers.

To assess the effectiveness of resizing-based defense, we varied the downsampling ratio within the range $r\in \{0.9, 0.8,\ldots,0.1\}$. As shown in Table 12, we observe that:

1. Resizing proves to enhance the tracker's accuracy under various attacks.
2. The efficacy of this enhancement varies depending on the attack type. Resizing significantly mitigates the impact of the SPARK attack, elevating precision from 69.8 to 83.9, but exhibits limited effectiveness against the RTAA, where precision increases modestly from 32.7 to 49.
3. The influence on RTAA remains constrained as precision increases from 32.7 to 49.3. Gradually increasing $r$ improves precision under RTAA but has adverse effects on precision in clean data and IoUAttack scenarios.
4. In comparison to the resizing method, LRR consistently improves tracker precision across all attacks, showcasing a noteworthy advantage while maintaining a high score on clean data.

Regarding compression, we utilize JPG compression for image reconstruction, adjusting compression qualities with $q\in {0.98, 0.96, 0.94, 0.92, 0.90 }$. The results are presented in Table 13, and the following observations are made:

1. Compression with a high-quality requirement shows limited impact on various attacks.
2. With a decrease in compression quality, precision increases across different attacks, underscoring the efficacy of compression as a defense mechanism against adversarial tracking.
3. The improvements achieved by compression under attacks are limited and fall short of the results obtained with LRR.

To address the concerns, we have added the above discussion in the supplementary material (Appendix A.8) with explanations in section 4.2.

Note that, in the original submission, we have involved the adversarial training (AT)-based methods as defense baselines in Section 4 and Table 1, which include three variants, i.e., $AT_{FGSM}$, $AT_{PGD}$, $AT_{CSA}$. We have introduced the three methods in the subsection "Defence baselines" on page 6-7.

Table 12: Comparing DiffPure (Nie et al. 2022) with LRR when defending CSA and RTAA.

Table 13: Comparing DiffPure (Nie et al. 2022) with LRR when defending CSA and RTAA.

Thanks for the comments. In our original submission, we have provided the visualization results of correlation maps and images before attack (i.e., clean examples), after attack (adversarial examples), and after reconstruction (defense examples) in the supplementary material (i.e., Appendix A.2) to validate our performance. To further address the concerns, we add more visualization results with higher resolutions in the supplementary material (i.e., Appendix A.2) with Figure 7, Figure 8, and Figure 9. Moreover, we have added corresponding explanations in the main submission (Sec. 4.1).

Thank you for your valuable suggestions. We've diligently proofread the entire submission, rectifying some typos and grammar errors. Due to space constraints, certain content is included in the supplementary material. We have integrated these details with our main paper as a whole, appropriately citing the results to ensure readers are aware of the relevant information in the supplementary material.

Thank you for your comment. We have configured the reconstruction range to be the search region rather than the entire image, significantly reducing the time costs. This is also aligned with the existing trackers' pipeline. To avoid confusion, we have revised the `Other details.' subsection in Sec. 3.4.

In our original submission, we have provided and compared the visualization results of correlation maps and images before the attack (i.e., clean examples), after the attack (adversarial examples), and after reconstruction (defense examples) in the supplementary material (i.e., Appendix A.3) to validate our performance. To address the concerns, we added more visualization results with higher resolutions in the supplementary material with Figure 7, Figure 8, and Figure 9. Moreover, we have added corresponding explanations in the main submission (Sec. 4.1).

Thank you for your valuable suggestions. In response, we have incorporated an experiment in Section 4.2, utilizing the recently proposed diffusion-based reconstruction method known as DiffPure (Nie et al. 2022) to assess its efficacy against adversarial tracking attacks. We contend that DiffPure may not be optimal for tracking defense due to its iterative denoising nature and time-consuming characteristics, making it less adaptable to real-time trackers. In our initial submission, we addressed the primary concerns associated with DiffPure in the related work section. To provide further clarification, we have introduced a new subsection in the supplementary material (Appendix A.7) to elaborate on this aspect. Additionally, a discussion has been included in Sec. 2 for a more comprehensive coverage.

Feasibility of using diffusion for tracking defense. We explore the efficacy of the recently developed diffusion-based adversarial defense method, DiffPure (Nie et al., 2022), for tracking defense. Specifically, we apply DiffPure to safeguard against three attacks—RTAA, IoUAttack, and CSA—based on the SiamRPN++ Res50 tracker. In our empirical study, we use DiffPure's default parameters for defense but vary the number of iterative time steps (i.e., T=1, 10, 50).

Table 11 illustrates that the three DiffPure variants enhance the precision of the tracker under different attacks, albeit to a lesser extent compared to our approach, LRR. Notably, DiffPure(T=50) is 86.9 times slower than LRR, requiring an average of 3391 ms for each frame, rendering it nearly impractical for tracking tasks. Even with a reduced time step to 1, DiffPure speeds up to 146 ms per frame, still 3.7 times slower than LRR. It is essential to note that the default DiffPure configuration sets T=100 time steps for purification, which is impractical for tracking tasks due to time constraints. In conclusion, further investigation is needed to understand the potential of leveraging diffusion for tracking defense.

Table 11: Comparing DiffPure (Nie et al. 2022) with LRR when defending RTAA, IoUAttack, and CSA.