LoRATv2: Enabling Low-Cost Temporal Modeling in One-Stream Trackers

Thank you for your valuable feedback and insightful questions. We provide our responses below.

---

Q1: [Ablation on LoRA Rank] The choice of LoRA rank is reported without an ablation study to justify the selected rank size. How do different LoRA rank values impact performance?

A1: Thanks for the comment. Our choice of LoRA rank $r=64$ was inherited from the hyperparameters used in the original LoRAT.

To verify if this is the suitable choice for the LoRATv2 framework, we conducted an ablation study on both our phase-1 (LoRATv2-B-224) and phase-2 (LoRATv2-B-378) models with various LoRA ranks. The results are presented below:

The results show that $r=64$ offers the best trade-off between accuracy and parameter efficiency for our model.

We will include the above analysis and results in revision. Again, thanks!

---

Q2: [Details on KV Cache] The discussion of the KV cache is insufficient. What is the maximum cache capacity, how much memory does it consume, and does caching multiple frames lead to bottlenecks or performance degradation?

A2: Thanks and we agree that a thorough explanation of the KV cache's practical impact is essential.

• Cache Capacity and Memory Consumption: The memory footprint of the KV cache is modest. For instance, caching the template and one past search frame in our LoRATv2-B-378 model consumes only 54.4MB, and for the larger LoRATv2-L-378 model, it is 145.6MB. This constitutes a small and manageable fraction of total GPU memory, which is negligible for most tracking scenarios. The cache capacity is not a hard limit and can be extended to retain more historical frames.

• Bottlenecks and Performance Degradation: Caching multiple frames does not lead to significant bottlenecks or performance degradation. This is for two primary reasons:

  1. Optimized Kernels: We leverage modern fused attention kernels (e.g., FlashAttention), which are specifically designed to handle such memory access patterns efficiently. These kernels effectively overlap memory operations (i.e., fetching the KV cache from memory) with computation, thereby hiding most of the memory access latency.
  2. Net Computational Gain: The KV cache provides a substantial net computational gain. Because attention is causal, embeddings for past frames are immutable—they are not influenced by any subsequent frames. They can be computed once, cached, and reused without ever needing to be re-encoded. This massive saving in computation far outweighs the minor cost of memory access.

• Performance with More Frames: To empirically demonstrate that our framework scales gracefully, we progressively extended our LoRATv2-B model to incorporate up to three past search frames. As shown in the table below, adding more frames to the cache consistently improves tracking accuracy while only minimally impacting efficiency, confirming that caching does not cause performance degradation.

These results confirm that not only does caching more frames avoid bottlenecks, it also actively improves performance with a very graceful degradation in FPS. We will provide these details about the KV cache by including the above analysis in revision. Again, thanks!

---

Q3: [More Ablations for FWCA and SSLA] The Phase 1 ablation experiments do not adequately demonstrate the effectiveness of FWCA and SSLA. Could you please supplement with an experiment involving standard self-attention + SSLA?

A3: Thanks for this thoughtful comment. This experiment was conducted and included in our appendix but omitted from the main text due to space limitations. To provide a comprehensive view, we present the full ablation table below and will ensure the designs for Phase 1 and 2 in Table 3 are consistent in the revision.

With standard Full Self-Attention, the bidirectional self-attention are symmetric, so moving from a Shared LoRA to SSLA provides only a marginal benefit. However, the combination of FWCA and SSLA not only yields comparable or better accuracy but also dramatically improves efficiency (e.g., 249 vs. 163 FPS for the -224 model). For the two-search-frame model, the benefits are even clearer, with our final design (FWCA+SSLA) substantially outperforming all other variants in both accuracy and efficiency.

We will include more detailed analysis in revision. Again, thanks!

---

Q4: [Performance of Phase 1 Model] What level of performance can be achieved using only the Phase 1 model in low-compute scenarios?

A4: Thanks for this insightful comment. As suggested, to further demonstrate its effectiveness in low-compute scenarios, we conducted an ablation of our method for CPU inference. Specifically we trained a LoRATv2-S-224 variant by accordingly adopting a smaller ViT-Small backbone. For a fair and direct comparison against leading lightweight trackers, all models were re-evaluated on the same consumer-grade CPU (Intel Ultra 7 265k).

The results show that our architecture also can be served as a highly competitive option within the domain of efficient visual tracking in low-compute scenarios. We will include the analysis in revision. Again, thanks!

[1] Kang, B., Chen, X., Wang, D., Peng, H., & Lu, H. (2023). Exploring Lightweight Hierarchical Vision Transformers for Efficient Visual Tracking. In: ICCV.

[2] Cui, Y., Song, T., Wu, G., & Wang, L. (2023). MixFormerV2: Efficient Fully Transformer Tracking. In: NeurIPS.

We appreciate your positive assessment and helpful suggestions. We address your points below.

Q1: [Fresher Baselines in Table 1] Table 1 needs fresher baselines.

A1: Thank you for the suggestion. We will update main comparison table (Table 1) in the revision by including more recent state-of-the-art trackers, as shown below:

[1] Xu, C., Zhong, B., Liang, Q., Zheng, Y., Li, G., & Song, S. (2025). Less is More: Token Context-aware Learning for Object Tracking. In: AAAI.

[2] Li, X., Zhong, B., Liang, Q., Li, G., Mo, Z., & Song, S. (2025). MambaLCT: Boosting Tracking via Long-term Context State Space Model. In: AAAI.

[3] Liang, S., Bai, Y., Gong, Y., & Wei, X. (2025). Autoregressive Sequential Pretraining for Visual Tracking. In: CVPR.

[4] Zhao, J., Chen, X., Yuan, Y., Felsberg, M., Wang, D., & Lu, H. (2025). Efficient Motion Prompt Learning for Robust Visual Tracking. In: ICML.

---

Q2: [OSTrack Speed Anomaly in Table 2] Although OSTrack shows lower MACs than LoRATv2, its FPS is lower. Explain this anomaly.

A2: Thanks for this careful comment. The discrepancy between theoretical MACs and practical FPS arises primarily from architectural differences affecting hardware utilization. OSTrack's "Early Candidate Elimination" strategy, while reducing MACs, introduces sequential logic and conditional branching that are less amenable to the massive parallelization of modern GPUs, thus increasing latency. And in our testing, it requires higher-precision (float32) arithmetic for stable performance, which is inherently slower. Furthermore, its convolutional head introduce higher FLOPs compared with our MLP-based head.

LoRATv2's design is highly parallelizable and optimized for fused GPU kernels, leading to higher throughput (FPS) despite the slightly higher MAC count.

We will include the above clarification in revision. Thanks again!

---

Q3: [Minor English Issues] The term “state-of-the-art” appears in different forms.

A3: Thank you for pointing this out. We will carefully proofread the entire manuscript to ensure consistent terminology in the whole manuscript.

Thank you for your detailed and constructive feedback. We address your questions and concerns below.

Q1: [Comparative analysis of training strategies] Provide a comparative analysis of direct multi-frame training versus the proposed two-stage strategy, covering quantitative results (accuracy/FPS) and resource consumption (GPU hours/memory) to demonstrate the necessity of phased training.

A1: We appreciate the reviewer's valuable suggestion. To highlight the necessity and advantages of our progressive training strategy, we compare it with a standard "from-scratch" multi-frame training baseline. All methods are based on LoRATv2-B-378, the only difference lies in the training strategy.

In this experiment:

• "One-Phase (1x)": Direct two-search-frame training for 170 epochs.
• "One-Phase (2x)": Doubles to 340 epochs.
• "Two-Phase (1x+1x)" is our proposed method: 170 epochs for the single-search-frame model, followed by an additional 170 epochs for the two-search-frame model.

The results clearly validate the advantages of our approach:

1. Higher Accuracy: Our two-phase strategy achieves 74.3% SUC on LaSOT, significantly outperforming the from-scratch model (71.3% SUC for "One-Phase (2x)"). This highlights the effectiveness of the curriculum-like learning process, which guides the model evolving from a simpler to a more complex task.

2. Greater Efficiency: Compared to the "One-Phase (2x)" baseline, our method requires substantially less training time (19 hours vs. 32 hours) and consumes less peak GPU memory (15.6 GB vs. 17.5 GB), primarily because fewer parameters are trainable in the second phase.

We will include this comparative analysis in the revision. Again, thanks!

---

Q2: [Scalability of SSLA] Provide a performance evaluation with more frame inputs to substantiate the scalability and efficacy of Stream-Specific LoRA Adapters in large-scale scenarios.

A2: We appreciate this insightful question regarding the scalability of our framework.

To empirically validate the scalability, we progressively extended our LoRATv2-B-378 model to incorporate up to three past search frames. Each new frame was managed by a new, progressively trained SSLA module.

The results are presented below:

This experiment reveals two key advantages of our approach:

1. Consistent Performance Gains: Tracking accuracy improves consistently as more frames are added. This demonstrates that our model, guided by the specialized adaptations from SSLA, can effectively leverage longer temporal history for more robust predictions, with particularly notable gains on the challenging VastTrack dataset.

2. Computational Efficiency: The computational cost grows very slowly. Scaling from one to three past search frames increases MACs by only ~9% (from 81G to 88G), with a minimal impact on FPS. This high efficiency stems from our architecture: the most computationally intensive ViT layers (Attention and MLP linear projections) process only the newest search frame's tokens. The complexity of the frame-wise causal attention mechanism grows linearly with cached frames, and its contribution to total MACs is minor.

We will include the above analysis and results in reivions. Thanks again!

---

Q3: [Clarity of “type emb.”] The "type emb" in Figure 2 is from LoRAT v1 but is not mentioned in the revisiting section.

A3: Thanks for this careful comment. We will add a clear explanation of this component in the revision.

---

Q4: [Clarity of Table 3] Table 3 conducts ablation experiments on model design. First, as a personal suggestion, if the final model design is used as the baseline for the experiment, the Variants column should use “-” to indicate changes in model design instead of “+”. For example, “+ w/o SSLA” should be “- SSLA”. Second, the ablation designs for Variants in phases 1 and 2 should be as consistent as possible.

A4: Thanks for suggestions. We'll update Table 3 to "-" notation and ensure Phase 1/2 consistency.

---

Q5: [Originality of FWCA and KV Cache] The core contributions are primarily engineering-driven enhancements rather than theoretical innovations: Frame-wise causal attention: While effective, this mirrors prior autoregressive designs and shares operational similarities with S-MAM in MixFormer. Key-value caching: Positioned as an implementation optimization (also seen in MixFormer's public codebase) rather than a novel methodological contribution.

A5: Thanks for this insightful comment.

Our Frame-Wise Causal Attention (FWCA) builds on autoregressive principles to enable efficient temporal modeling without recomputing historical frames. However, autoregressive models are prone to error accumulation[1]: early prediction mistakes propagate and amplify over time, especially during early training phases. To specifically mitigate this issue, we introduced novel Stream-Specific LoRA Adapters (SSLA) and the two-phase progressive training strategy, which together ensure stable convergence and robust temporal modeling.

This design sets our approach apart from methods like S-MAM in MixFormer. While S-MAM prunes target-to-search cross-attention and attention among target templates, effectively treating multiple templates as a reference bank for localization, our Frame-Wise Causal Attention establishes a genuine autoregressive history. By sequentially processing frames, it captures the target's evolving appearance and state, which strengthens tracking robustness and paves the way for future advancements in motions-aware or occlusions-aware trackers. As evidenced in our experiments, this design leads to superior performance compared to S-MAM.

Although KV caching serves as an known optimization to accelerate autoregressive inference, akin to implementations in other codebases, our key innovation lies in integrating it with SSLA and progressive training to address error accumulation.

Again, we thank the reviewer for these helpful comments. We will include the above explanations and more discussions in revision to clarify the difference of our method with other methods. Thanks!

[1] Yin, T., Zhang, Q., Zhang, R., Freeman, W. T., Durand, F., Shechtman, E., & Huang, X. (2025). From Slow Bidirectional to Fast Autoregressive Video Diffusion Models. In CVPR.

---

Q6: [Clarity of Model Variants] The "Model Variants" section (4.1) is unclear. The model's input settings for different phases and resolutions should be clearly specified.

A6: We apologize for the lack of clarity. We will revise it as follows:

• LoRATv2-B/L-224 (Phase 1 Models):
  • Backbone: ViT-Base/ViT-Large
  • Template ($\mathbf{z}$): $224 \times 224$
  • Search Region ($\mathbf{x}^1$): $224 \times 224$
• LoRATv2-B/L-378 (Phase 2 Models):
  • Backbone: ViT-Base/ViT-Large
  • Template ($\mathbf{z}$): $224 \times 224$
  • Past Search Region ($\mathbf{x}^1$): $224 \times 224$
  • Current Search Region ($\mathbf{x}^2$): $378 \times 378$

---

Q7: [Parameters in Efficiency Table] In Table 2, the parameters of LoRAT are consistent across different resolutions, but those of LoRATv2 are not. Explain the reason for this.

A7: Thanks for this careful comment. The difference stems from a core design choice:

• LoRAT uses a single set of LoRA adapters whose weights are shared between the template and search streams. Thus, changing input resolution does not alter the number of trainable parameters.
• LoRATv2, with our proposed SSLA, uses dedicated LoRA adapters for each stream. The "-378" models introduce an additional stream for the second search frame ($\mathbf{x}^2$), which requires its own new LoRA adapter and a new prediction head. This is the source of the increased parameter count for our larger, multi-search-frame models.

We will add a clear explanation of this point in Section 4.2 when discussing the efficiency comparison. Again, thanks!

---

Q8: [Convergence Failure] Explain the reasons for the failure of variant ② in Table 3 (FWCA with Shared LoRA) to converge.

A8: The convergence failure is caused by the functional asymmetry introduced by FWCA, which cannot adequately model with a simple shared LoRA. If a shared LoRA is tasked with learning two functionally distinct adaptations using a single set of weights, it woule lead to unstable optimization. Our SSLA adapt such asymmetry in network structure.

To validate this, we conducted a detailed ablation study:

The results are revealing:

• With symmetric Full Self-Attention, all methods converge, as Shared LoRA is sufficient.
• With asymmetric FWCA, Shared LoRA fails entirely. Full fine-tuning can adapt with the asymmetry but results in suboptimal performance.
• Only the combination of FWCA and SSLA both converges reliably and achieves the best performance, demonstrating that SSLA is an essential component of our framework.

We will include the above clarification and results in revision. Again, thanks for this comment!