Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking

We thank the reviewer for recognizing the novelty and strong results of our approach, as well as for the high ratings on the soundness and presentation of our manuscript. The reviewer’s comments inspired the paragraph on Efficiency Analysis and Comparison with Other Sequence Models in the General Comment to this rebuttal, which we will also include in the camera-ready version. We here address the weaknesses raised in the review.

---

Computational Complexity

As detailed in the General Comment (see Efficiency Analysis), we have analyzed the computational complexity of our approach by evaluating the following metrics:

1. Peak memory consumption during training,
2. Throughput and latency during inference, and
3. The number of model parameters.

This analysis compares SambaMOTR with the baseline MeMOTR and includes the integration of different sequence models within the Samba framework, as suggested by the reviewer. For detailed results and insights, please refer to the General Comment.

---

Comparison with Other Sequence Models

We thank the reviewer for emphasizing the importance of comparing Samba with alternative sequence models. Since Samba is inherently flexible regarding the underlying sequence model, we included comparisons with a representative state-space model (S4) and a state-of-the-art RNN-based approach (xLSTM).

As reported in the General Comment (see Performance Comparison of Samba with Other Sequence Models), our analysis shows that Mamba-powered Samba outperforms both xLSTM and S4 in terms of tracking performance (HOTA). Additionally, Mamba is more parameter-efficient, requiring significantly fewer parameters (42.0M vs. 49.4M for xLSTM). These results further highlight the effectiveness and efficiency of Mamba as the sequence model within the Samba framework.

We appreciate the reviewer’s positive feedback on the motivation of our work, its strong performance, the extensive ablation studies, and the clarity of our manuscript. We here address the remaining weaknesses and questions raised by the reviewer. Regarding qualitative comparisons to prior work, while the rebuttal format does not permit inclusion of such results, we commit to providing additional qualitative comparisons in the camera-ready version, specifically focusing on tracking under occlusion.

---

GPU Memory Requirements

We acknowledge that our claim in line 137 lacked supporting evidence, and we thank the reviewer for highlighting this. In particular, in line 137 we claimed that our efficient training recipe to learn from longer sequences allowed "improving the tracking performance while maintaining the same GPU memory requirements as previous methods."

As suggested, we have conducted an efficiency analysis of SambaMOTR and combined it with WHQW's suggestions to analyze Samba with different sequence models. Reported in the General Comment - Efficiency Analysis, this analysis includes metrics such as peak memory consumption, throughput, latency, and parameter count. Regarding GPU memory, our results show that SambaMOTR using Mamba without synchronization has the same peak memory consumption as MeMOTR. Introducing synchronization increases the peak memory by only 0.5 GB.

More specifically, our claim in line 137 refers to the efficient training strategy we use for learning from longer sequences. Thus, in the following table, we report the peak memory consumption during training when training on 5 frames vs. when training on 10 frames with our strategy. This strategy allows training on longer sequences without increasing peak memory consumption. Moreover, it results in improved tracking performance, as evidenced by the HOTA scores.

---

Disentangling Performance Improvements from Model Capacity

We understand the reviewer’s concern regarding the source of our performance improvements. The reviewer mentioned that "it is worth asking whether efficiency gains enable higher capacity models, which lead to improvements or whether the SSMs formulation provides better biases for the tracking problem." To address this, we compared SambaMOTR’s parameter count to that of MeMOTR, considering different sequence models, state dimensions, and the presence or absence of synchronization.

As detailed in the General Comment - Efficiency Analysis, our findings indicate the following:

1. Mamba introduces fewer parameters than the heuristics-based propagation head of MeMOTR while achieving better performance.
2. The additional parameters introduced by synchronization are proportional to the hidden state dimension. For example, using a hidden state with four channels results in a dimensionality of 1024x4=4096 for the synchronization module’s feedforward network. This configuration increases the parameter count from 42.0M (baseline without synchronization) to 54.6M. Keeping the state dimension to 1 significantly lowers the parameter cost for synchronization, increasing the total parameter count to only 43.6M.

Crucially, even without synchronization, our baseline model with 42.0M parameters outperforms MeMOTR. Thus, the addition of synchronization and the choice of hidden state size can be adapted to match the practitioner's computational requirements.

As the reviewer already mentioned, we hope that this analysis provides valuable guidance for future research.

---

Comparison of SambaMOTR on Top of YOLO-X

While we agree with the reviewer that Samba can theoretically be extended to the tracking-by-detection paradigm by replacing the Kalman filter to jointly model motion and appearance trajectories, we believe that this analysis is better suited for a separate study. We are actively working on this extension.

Since the current paper targets end-to-end tracking-by-propagation, we maintain that the most meaningful comparison is between MeMOTR and SambaMOTR.

We here address the weaknesses.

---

Interpretation of System A and Control B Matrices

We agree that a clearer explanation of the system $A$ and control $B$ matrices in the context of object tracking will benefit readers. Using the notation from Eqs. 3a-3c:

• For a tracklet $i$, the observation $x_t^i$ at time $t$ is a joint representation of the object’s current location and visual appearance.
• The hidden state $h_{t-1}^i$ represents the object’s history, including its past motion, appearance, and pose changes.

In the memory update equation (Eq. 3a):

$$h_t^i = \mathbf{\bar{A}}^i(t)h_{t-1}^i + \mathbf{\bar{B}}^i(t)x_t^i,$$

we can disentangle it into:

1. $\mathbf{\bar{A}}^i(t)h_{t-1}^i$: Represents system dynamics, predicting the expected future hidden state (where the object will move and how it will look like) based only on past motion and appearance.
2. $\mathbf{\bar{B}}^i(t)x_t^i$: Incorporates the current observation to correct the predicted hidden state.

Thus, $\mathbf{\bar{A}}^i(t)$ represents the tracklet's temporal dynamics, while $\mathbf{\bar{B}}^i(t)$ refines these dynamics using the latest detection. This dual mechanism ensures robust modeling of both long-term dependencies and immediate corrections. We will include this clarification in the camera-ready version.

---

Tracking-by-Detection (TbD) vs. Tracking-by-Propagation (TbP)

We argue that the distinction between TbD and TbP is significant, both conceptually and in practice. To illustrate:

• In TbD, the detector operates independently for each frame: $y_t = f(I_t)$, where $f()$ is the detection function. Detection results $y_t$ are post-processed with a downstream association step.
• In TbP, the detector is conditioned on previous detections: $y_t = f(I_t, g(y_{t-1}))$, where $g()$ is the propagation module that propagates prior outputs. This conditioning is done by propagating the output embeddings corresponding to detected objects at time $t-1$ and appending them ($g(y_{t-1})$) at time $t$ to the list of detection queries before feeding them to the transformer decoder at time $t$. This conditioning fundamentally changes how the detector operates during both training and inference, creating a distinct paradigm. In particular, TbP methods can rely on end-to-end training over videos, bridging the training-inference gap and leveraging temporal dependencies.

We will improve the clarity of this distinction in the paper.

---

Comparison with Tracking-by-Detection (TbD)

Based on the assumption that TbD and TbP are not inherently different, the reviewer points out:

"Then it should be fair to compare with TBD methods, and the performance still falls short behind classic motion models as reported in Table A."

While we hope to have clarified in the previous section that the difference between the two families of approaches is significant, we acknowledge the reviewer’s observation that TbP methods fall short on MOT17 compared to TbD (Table A).

However, this limitation stems from the training requirements of TbP, not an inherent inferiority of the approach.

TbP methods rely on end-to-end training over videos, bridging the training-inference gap and leveraging temporal dependencies. In contrast, TbD methods use independent frames for training, allowing them to benefit from large-scale datasets like CrowdHuman. Since MOT17 is a small dataset with only 7 annotated videos (line 897), the scarcity of video information limits TbP training effectiveness.

To mitigate this, we augmented the MOT17 training set with pseudo-videos generated by applying multiple data augmentations on individual CrowdHuman images. While this improves results, pseudo-videos cannot fully replicate real-world dynamics, leading to performance gaps due to the mismatch between the end-to-end training and the autoregressive inference. This limitation highlights an important area for future research, and we hope the reviewer appreciates our transparency in acknowledging it (Sec. C.1), as recommended by ICLR guidelines.

---

Motivation and Novelty

While the reviewer questions the motivation and novelty of our approach, we believe these aspects are strongly supported by our contributions and results:

1. Motivation: Current end-to-end tracking methods fail to model long-range dependencies and object interactions effectively. Samba addresses this gap by introducing learnable set-of-sequences models in TbP, as appreciated by other reviewers and confirmed by our performance improvements.
2. Novelty: Samba is not a mere combination of MOTR and Mamba. Samba is flexible to other sequence models (see General Comment). Additional key contributions of our submission (see ablations) include synchronization, MaskObs, and efficient training on longer sequences. These components are indispensable to our framework and collectively enable its strong performance.

Hoping this rebuttal addresses the concerns, we remain available for discussion.

We thank the reviewer for praising the extensive experiments and for providing constructive criticism. We here address the questions raised in the review.

---

Does the state space model operate on the coordinate domain (bounding boxes) or the visual domain (pixel values)?

The reviewer raises an important point: if Samba operated solely on the coordinate domain, auto-regressed output boxes could not be refined to accurately fit subjects without visual feature input.

Our Samba module operates on both the coordinate and visual domains. As explained in Sec. 4.3 - Query Propagation with Samba, we construct the inputs to the sequence model by combining the bounding box coordinates and query embeddings outputted by the detector’s transformer decoder. Specifically, we sum the positional embedding of the predicted bounding box coordinates with their corresponding visual representation (query embedding).

To validate this design, Table D (in the Appendix) compares the effects of including or excluding the positional embedding. Results show that combining coordinate and visual representations consistently outperforms using visual features alone. This illustrates how our propagation module can simultaneously model object trajectories in both coordinate and visual spaces.

We hope that this answers the reviewer's question, potentially clarifying that one of the novelties also lies in how our propagation module can simultaneously model object trajectories both in the coordinate and visual space.

Thank you for your efforts in clarification. I hope to see these details discussed in the revision. In the light of this, I have increased my rating.

Considering my suggestion, more qualitative comparisons show that the Samba autoregressive model performs better than memory/motion models should be provided to visually substantiate the claims.