Is Multiple Object Tracking a Matter of Specialization?

W1 - Comparison with other methods in zero-shot

We herein test ByteTrack and other tracking-by-detection methods in a zero-shot setting from MOT17 to PersonPath22. The results on PersonPath22 are as follows:

Results indicate that PASTA leads to remarkable improvements compared to the other query-based end-to-end approach (i.e., MOTRv2), even though they are both outperformed by the tracking-by-detection methods (such as ByteTrack). To be more comprehensive, PASTA remains competitive in terms of association performance (IDF1), but it yields weaker detection capabilities. Such a trend does not surprise us and is in line with what occurs in the more standard evaluation, where fine-tuning on the target dataset is allowed. Indeed, tracking-by-detection approaches are generally more robust than those based on end-to-end learning, so much so that it is an established practice of researchers to present the results in separate parts of a table [10, 43, 48], to deliver an apple-to-apple comparison.

Concerning the zero-shot setting in our work, we conclude that existing tracking-by-detection trackers are more robust to domain shifts. In these approaches, the only part potentially subject to shifts is the detector (e.g., YOLOX). Instead, the motion model (e.g., Kalman Filter) and the association strategy [3, 46] are almost parameter-free procedures that are less affected by domain shifts for construction, as their design reflects strong inductive biases about human motion. For such a reason, it is our belief that the problem of domain shift in Multiple Object Tracking (MOT) should be primarily addressed in parametric approaches such as deep neural networks. For this reason, our research question focuses on query-based trackers (e.g., MOTRv2) that learn entirely from data. Our final goal is to enhance these trackers, as their end-to-end nature results can lead to challenges during domain shifts. We will include such a discussion in the final manuscript.

W2 - Detection Thresholds used in Tab 1

We evaluated ByteTrack on MOTSynth using the default values provided in the public ByteTrack repository, specifically a minimum confidence score (min_score) of 0.1 and a track threshold (track_thresh) of 0.6. Below, we present the results for different values of these thresholds:

The results are close, with a slight improvement when reducing the track_thresh to 0.3 or 0.4, while min_score of 0.1 remains optimal. We will report these updated results in the final version, along with more details about the parameters used.

W3 - Choice of attributes

We selected the attributes reported because we believe they are the most generic and applicable to various scenarios. However, users are not limited to using these specific attributes; PASTA is fully customizable to suit different needs. For instance, if users know their system will always be employed outdoors, they could omit the indoor/outdoor attributes and add a good/poor weather attribute instead. This flexibility allows PASTA to be tailored to specific requirements, enhancing its adaptability and effectiveness across various applications. This adaptability ensures that users can optimize the trakcer for their unique contexts without needing extensive retraining or adjustments.

Furthermore, PASTA's modular nature means that users can easily integrate additional modules to handle these changes as new attributes become relevant or new scenarios emerge. This future-proofs the system and allows it to evolve alongside technological advancements and shifting user needs. For example, attributes such as traffic density or driving side could be added in an automotive setting to improve performance. The ability to customize and extend PASTA ensures it remains a robust and versatile tool for a wide range of tracking applications.

W4 - Omitting LoRA

Omitting LoRA (or other Parameter-Efficient Fine-Tuning techniques) in PASTA would significantly impact its efficiency in scenario-specific adaptation. Without these techniques, each attribute would require a fully fine-tuned model, which poses several challenges, especially memory constraints [12,19]. Firstly, storing a separate model for each attribute is highly storage-intensive. For instance, a PASTA module is approximately 5MB, whereas the full model exceeds 350MB. With 12 attributes, the total storage requirement for PASTA would be 410MB (350MB + 12 x 5MB). In contrast, storing 12 fully fine-tuned models would require around 4.2GB (12 x 350MB), representing a tenfold increase in storage needs. Additionally, adapting an entire model to each specific condition is more time-consuming than using LoRA, as it involves optimizing a more significant number of parameters. This adaptation process must be repeated for each attribute, making it both impractical and costly. Moreover, fully fine-tuning a transformer-based architecture demands more data than a parameter-efficient approach.

Thanks for your response. The authors address my main concerns.

W1/Q1 - On the selection of modules

In our work, we use a Domain Expert to select attributes, a practice that is not far from reality. For example, in fixed-camera scenarios, the mounting perspective is known and whether it will be indoors or outdoors. Lighting can be easily measured with a simple computer vision algorithm, and occupancy can be determined by counting detections. Therefore, the assumption of using a Domain Expert is both reasonable and practical.

W2/Q2/L2/L3 - On the evaluation on other domains

We argue that MOT17 and PersonPath22 represent practical, real-world surveillance applications. Unfortunately, due to time constraints, we cannot perform other extensive further evaluations during the rebuttal phase.

W3/Q4 - On the computational cost

We compare the GPU memory and time requirements of full fine-tuning versus our approach on the MOTSynth dataset. Our method reduces GPU memory requirements from 13GB to 8.25GB, a reduction of over 35%. This significant decrease is due to the lower number of parameters updated by the optimizer: 42M parameters for standard fine-tuning versus 15M for our approach. Regarding timing, we achieved a reduction from 0.6 seconds per iteration to 0.55 seconds per iteration on average, which is approximately a 10% improvement. Overall training time is reduced from around four days to about 3.5 days on average.

Additionally, our model's adaptability to various scenarios without fine-tuning significantly reduces deployment time across diverse domains.

W4/Q3 - Synergy between modules

We kindly refer the reviewer to Tables 5 and 6 and Figure 2 for an analysis of the synergy between different modules. We explore various methods of aggregating and selecting modules. Selecting opposite modules (e.g., poor lighting when the scene has good lighting) results in worse performance than selecting the correct module. Specifically, Figure 2 demonstrates that adding the correct module one at a time monotonically increases both IDF1 and MOTA. We will make sure to highlight these experiments better in the final revision.

W5 - Module robustness

The experiments we conducted on MOTSynth, MOT17, and PersonPath22 (Tables 1 to 3) cover a wide range of scenarios, attributes, data volumes, and quality. MOTSynth includes nearly 1.5 million frames and 800 different scenes, PersonPath22 encompasses over 200,000 frames and 236 scenes, while MOT17 comprises 5,316 frames across 14 scenes. These datasets differ significantly in volume and quality, with MOTSynth being a synthetic dataset featuring high-quality images and annotations. PersonPath22 also boasts good image quality and accurate annotations. Conversely, MOT17, having been around longer, presents more challenges due to its relatively lower image quality.

Additionally, our zero-shot experiments demonstrate PASTA's strong generalization capabilities and its resilience to noise, confirming its robustness across diverse conditions and datasets.

Q5 - Future Work

The discussion about future work is indeed brief and could be expanded. One aspect that warrants detailed analysis is the routing technique. As suggested by the reviewer, employing reinforcement learning to select the most appropriate modules based on the current scene could be beneficial. Additionally, as suggested by 6tD5, expanding the modules to describe dataset-level characteristics, such as automotive versus pedestrian settings, would be an interesting avenue to explore. Another potential topic for future work involves applying additional domain adaptation techniques, such as GHOST and DARTH (as mentioned in response to oFC1), which could further enhance the adaptability of our approach. We will ensure that these future work discussions are thoroughly addressed in the final version of the manuscript.

L1 - Increase in complexity

Our framework introduces a set of parameters for each attribute, which is the only increase in complexity. Training and optimization of modules are conducted in parallel, closely aligning with a standard query-based tracker like MOTRv2. Validation requires only the merging of weights based on the scene to be evaluated. In the deployment phase, there is an additional requirement for selecting or weighting modules. This slight increase in complexity offers significant benefits. Specifically, this additional step eliminates the need to fine-tune the model in specific environments. Our approach provides a highly adaptable and efficient tracker by customizing the attributes to suit different scenarios.

L4 - Dependance on pre-train

Our framework aims to enhance the knowledge transfer of end-to-end trackers. While the performance of the PASTA framework does depend on pre-trained backbone networks and detectors, we specifically address scenarios where these pre-trained models may perform poorly. Our approach involves fine-tuning on these specific scenarios without altering the pre-trained weights, as we only train a small, disjoint set of parameters. This allows us to retain the benefits of the pre-trained models while improving performance in targeted areas. In our experiments, we compare classical fine-tuning, named MOTRv2-MS, with our modular approach, showing better control and selection of parameters for different scenarios.

L5 - Real-time performance Our approach does not introduce additional overhead compared to MOTRv2 during inference, so the frames per second (FPS) remain unchanged. Specifically, the YOLOX detector performs at 25 FPS, and MOTRv2 operates at 9.5 FPS with a 2080Ti GPU. When combining these two components, the overall speed is 6.9 FPS.

We thank the reviewer for appreciating the novelty and efficacy of our proposed approach. We will answer in the following the doubts of the reviewer:

W1 - Evaluation on MOT20

We decided not to evaluate our method on MOT20 since its variety is limited in terms of attributes, as it contains only four similar sequences specifically designed to be crowded. In contrast, the benchmarks we use—MOT17, MOTSynth, and PersonPath22—capture a wide range of attributes and scenarios. The lack of diversity implies that MOT20 as a benchmark might not be representative for assessing our attribute-based approach. For instance, MOT20 lacks scenarios with low or moving cameras. We report in the following the complete MOT20 attribute statistics.

W2 - On scenes as attributes

We chose "weak" (fine-grained) attributes to better align with fine-tuning approaches. However, it would be interesting to explore the addition of "strong" (coarse) attributes to represent completely different scenes or settings. For example, an attribute for automotive settings and another for pedestrian settings. We feel that the rebuttal phase is too short to do justice to such an evaluation, which requires extensive training and experiments. However, we greatly appreciate the reviewer’s suggestion and aim to carry it out in future work.

W3 - Table 1 parameters

In Table 1, we reported the total number of trainable parameters (15M for PASTA vs 42M for MOTRv2). Once trained, the total number of parameters in our approach is the same as MOTRv2, i.e., 42M. We will provide a clearer specification of the number of parameters in the final version of the manuscript and will include a better description in the table captions. We thank the reviewer for this suggestion.

As pointed out by the reviewer, a more detailed discussion about the computational costs and the relationship with other domain adaptation methods would enhance the quality of our manuscript. Below, we answer these questions and will integrate this information into the final revision.

W1/Q1 On the computational cost

We compare the GPU memory and time requirements of full fine-tuning versus our approach on the MOTSynth dataset.

Our method reduces training GPU memory requirements from 13GB to 8.25GB (for a batch size of 1), a reduction of over 35%. This significant decrease is due to the lower number of parameters updated by the optimizer: 42M parameters for standard fine-tuning versus 15M for our PEFT technique. Regarding timing, we achieved a reduction from 0.6 seconds per iteration to 0.55 seconds per iteration on average, which is approximately a 10% improvement. This is because most of the network parameters remain frozen. Overall training time is reduced from around four days to about 3.5 days on average.

Additionally, since our final model is easily adaptable to a wide variety of scenarios without further fine-tuning, the overall time requirement for deployment across diverse domains is sensibly less compared to standard fine-tuning.

Finally, our approach does not add any overhead during inference, aside from weight merging, which is negligible for stationary attributes, compared to MOTRv2, which maintains a speed of 6.9 FPS on a 2080Ti GPU.

W2/Q2 Relation with domain adaptation and zero-shot methods

We acknowledge that the number of works handling domain adaptation and MOT or zero-shot MOT is very limited. We will make a comparison of our approach with the most relatable in the following:

Domain Adaptation methods

To the best of our knowledge, no tracking-by-query method currently employs domain adaptation. However, we have identified two tracking-by-detection methods that utilize domain adaptation: Simple Cues Lead to a Strong Multi-Object Tracker (GHOST) [30] and DARTH: Holistic Test-time Adaptation for Multiple Object Tracking [29].

GHOST is a simple tracker that leverages detections from an off-the-shelf detector. Specifically, GHOST employs "on-the-fly Domain Adaptation," which involves updating the Batch Norm layer statistics during inference (similar to test-time adaptation techniques) in the layers handling reID features. There are several critical differences between GHOST and our approach. First, we only combine already trained modules. Second, the adaptation in GHOST is applied only to the reID module features, whereas our adaptation affects the entire network. Since our approach is end-to-end and query-based, it impacts the tracking and detection components.

DARTH employs a test-time adaptation (TTA) technique to adapt the model from a source dataset to a target one using a Knowledge Distillation approach. Each adaptation step requires three forward passes to compute all objective functions, making the process computationally and memory intensive. Additionally, DARTH performs offline TTA, adapting the model using the entire sequence before evaluating it, which is not comparable to our fully online approach. A key difference from our method is that DARTH's requirement of having the entire sequence available for adaptation makes it challenging for real-world use cases. In contrast, our approach only requires simple attributes of the target scene, eliminating the need for further training or adaptation during deployment.

Open-vocabulary methods

Recent advancements in zero-shot multiple object tracking demonstrate a significant shift towards open-vocabulary multiple object tracking, a specific setting where textual descriptions define new categories to track. In this context, we recognize the latest works. OVTrack is an open-vocabulary tracker based on Faster R-CNN, enhanced with CLIP distillation to learn open-vocabulary capabilities. It also employs a hallucination strategy using denoising diffusion models for robust appearance feature learning. Z-GMOT introduces a dataset comprising videos and textual descriptions of target attributes and a tracker that improves detection through advanced grounded language-image pretraining. OVTracktor is another open-vocabulary tracker that can detect and segment any category.

Our method, however, does not involve open-vocabulary and language models. Specifically, PASTA focuses on comprehensively transferring domain knowledge of end-to-end trackers with a modular approach. Unfortunately, such open-vocabulary methods are not easily comparable to our approach. They require additional text data during inference and produce results only on open-vocabulary datasets rather than classical MOT benchmarks like MOT17.

Thanks to the authors for addressing my concerns by providing more details about the practical gains regarding training time and memory requirements and broadening the related work. I will maintain the "accept" rating.

We thank the reviewer for the valuable questions and for having appreciated the originality and efficacy of the proposed approach.

W2 - Dataset statistics

We report the requested details in the following tables, which provide statistics on the employed datasets divided by per-sequence and per-frame attributes

W3 - Ablation of task vectors contribution

We agree that an additional ablation study on where to apply the task vectors might clarify which components are most critical for adaptation over scenario attributes. To investigate this aspect, we trained three versions of PASTA by omitting the task vectors from the encoder, decoder, or backbone, respectively. We will report these results computed on MOTSynth in the final manuscript:

The results indicate that not applying task vectors to the decoder significantly degrades detection and association metrics. We believe that this degradation can be explained by considering the crucial role of the decoder. The decoder must indeed gather information from detection, tracking, and proposal queries while simultaneously integrating visual information from the encoder. Consequently, not adapting the decoder prevents the architecture from effectively leveraging queries and visual cues. The encoder also contributes substantially, though to a lesser extent than the decoder, as it primarily refines and contextualizes visual features from the backbone. The backbone shows the smallest contribution.

Q1/W4 - Attributes availability during training

During training, it is important to identify the relevant attributes for the specific problem at hand. In our approach, the selection of attributes was primarily guided by common sense and our experience with the task. Also, we chose attributes that are generally applicable to most use cases. However, this selection serves as a proof of concept. In practical applications, the selection could be further refined to reflect specific characteristics of the problem under consideration.

For instance, in a surveillance scenario characterized by indoor cameras, the indoor/outdoor attribute may become unnecessary. Conversely, an attribute for weather conditions might be valuable in locations with variable weather.

The need for predefined attributes is easily addressable using automatic or semi-automatic classifiers. For example, an analysis of the brightness level can easily classify lighting conditions or a straightforward detector can be exploited to count objects of interest in the scene, thereby classifying crowd density.

W1/Q2 - Real-to-real (MOT17 → PersonPath22)

We report below the results of the model trained on MOT17 and evaluated in zero-shot on PersonPath22.

To provide a comparison, we train MOTRv2 on MOT17 and test their performance on PersonPath22. Our approach yields better results compared to fine-tuned MOTRv2, demonstrating that exploiting modules improves the model's generalization capabilities on new and/or real domains. We will include these findings in the final revision of the manuscript.

Q3 - Missing MOTRv2 full tuning in Tables 2 - 3

We agree that including fine-tuned MOTRv2 results would provide a better understanding of the zero-shot experiments. We primarily did not undertake it due to time constraints and computational overheads. Herein, we remedy this and report the results for Table 2 (MOT17):

Notably, our zero-shot approach performs similarly to its fine-tuned counterpart. We appreciate the reviewer highlighting this missing comparison, which strengthens our findings.

Regarding Table 3 (PersonPath22), we have included fully trained methods on PersonPath22 as reported in the PersonPath22 paper. We could not train MOTRv2 on this dataset primarily because the authors of the PersonPath22 did not release the YOLOX weights they used for the other methods listed in their table nor provide interpolation scripts for the training-set annotations. Since training MOTRv2 on PersonPath22 is not crucial for the PASTA evaluation, and given the abovementioned limitations, we opted to refrain from reporting the full tuning of MOTRv2.