Tracking and Understanding Object Transformations

We thank Reviewer 6G8j for their positive evaluation and for recognizing the motivation and novelty of our work. Below, we address the reviewer’s specific questions and concerns.

---

W1. High computational cost

We acknowledge the concern regarding computational cost and agree that the current speed limits real-time applications. However, we believe TubeletGraph’s unique capabilities provide significant value despite this limitation.

Value Beyond Real-Time Performance: TubeletGraph enables detecting and describing object state transformations beyond object tracking in videos. No existing methods perform both temporal localization and semantic description of the transformation in addition to state-of-the-art performance on object tracking under transformation. The combination of tracking and discovering, localizing and describing transformations from videos can be very useful for many applications: for example, producing training annotations on recorded demonstrations for robots, analyzing compliance and processes on the factory floor from videos, understanding animal behavior and development from camera traps and so on. In these applications, understanding and tracking object transformations is critical, and real-time processing is not needed.

Potential for Amortized Efficiency: While the spatiotemporal partitioning is presented to be dependent on input object masks for clarity, our actual implementation (available in supplementary code) treats partition construction as a single preprocessing step that is shared across all objects of interest. This design allows the computational cost to be amortized when tracking multiple objects simultaneously. After preprocessing, TubeletGraph operates at speeds comparable to the base SAM2 model.

Bottleneck Analysis: The primary bottleneck is in constructing the spatiotemporal partition, which requires independent SAM2 tracking for each identified spatial region. This process is inherently parallelizable. Furthermore, we envision that future end-to-end systems can track multiple objects without incurring cost linear in the number of objects in the video. While important for practical deployment, we consider this optimization direction orthogonal to our core contributions of tracking and describing object transformations.

---

W2. Use of a non-open source models for transformation reasoning and description

We further evaluate semantic correctness when GPT-4.1 is replaced with the open-source Qwen2.5-VL-7B-Instruct. We show comparable performance across both models below, with Qwen marginally underperforming GPT4.1 in terms of action verb description. For details regarding the experimental setup, please refer to our response to Reviewer jTZM, W1.

---

W3. What effects do occlusions have on the model?

This is indeed a very interesting question. During occlusion events, the model would generally add additional tubelets for the object that is temporarily lost. If the additional tubelet matches the semantic and proximity constraints, it will be incorporated in the tracked object. Since the proximity constraint relies on the three candidate masks predicted by SAM2, the base tracker’s internal candidate masks would set an upperbound for the recall under occlusion. In terms of the transformation, since the VLM only sees the two frames where the initial and resulting objects are visible, it will not be able to describe the occlusion event as a transformation.

Nevertheless, addressing the occlusion challenge with the aid of the VLM is an interesting direction for future works, and we will add discussions on this in the final manuscript.

---

W4. Why does ReVOS outperform on M3-VOS?

Indeed, ReVOS outperforms our approach on M3-VOS, but it significantly underperforms on other datasets with object transformations. According to its authors, they “assume SAM2 takes more training data similar to EPIC-kitchens [5] from which the VOST is constructed.” We share that belief that M3-VOS and DAVIS17 contain far less ego motion compared to VOST. As such, the lower performance of our approach on M3-VOS compared to ReVOS can be traced to the fact that our base SAM2.1 tracker is seeing a distribution shift, compared to the ReVOS tracker.

This also explains the differences on DAVIS17. Given very similar performances for TubeletGraph and base SAM2 models, we show that our method doesn't overfit to transformation scenarios / degrade performance when objects do not undergo transformation.

We will add discussions on this in the final manuscript.

---

W5. Limited temporal localization evaluation and confusing choices for varying threshold values

We acknowledge the concern. Due to the constrained rebuttal period, we are not able to expand the label set further, but we promise to extend the evaluation to the entire VOST val set in the camera ready.

The varying T values were chosen to see how much recall / precision we can gain by setting the proximity threshold to be low (0) and setting the semantic threshold to be relatively high (0.9) respectively. We found that increasing the proximity threshold does not lead to increasing precision, since SAM2’s internal candidate masks, which are what the proximity prior relies on, induce an upper bound. We will incorporate this in the final version.

---

W6. Semantic consistency brings little improvement upon ST partition and proximity constraint

This is indeed a good observation. The semantic constraint does provide small, but consistent gain across all metrics when included. There are cases where semantic cases help, for instance when the tracker latches on to hands or other actors that begin interacting with the tracked object. In these cases, the proximity prior alone is not sufficient to reject the negative candidate. We will provide more example videos / discussion in the final version.

---

Q1. Details for fine-tuning SAM2 on VOST

No, we do not annotate anything further. SAM2.1(ft) is directly finetuned on the existing annotations in VOST. SAM2.1 finetuning does not require the transformations to be explicitly annotated. The finetuned model SAM2.1(ft) tracks the objects in a single binary mask and does not output the transformations (unlike our approach).

We thank reviewer rGHV for their insightful comments and suggestions. We offer details to address the concerns below:

---

W1. High computational cost

We acknowledge the concern regarding computational cost and agree that the current speed limits real-time applications. However, we believe TubeletGraph’s unique capabilities provide significant value despite this limitation.

Value Beyond Real-Time Performance: TubeletGraph enables detecting and describing object state transformations beyond object tracking in videos. No existing methods perform both temporal localization and semantic description of the transformation in addition to state-of-the-art performance on object tracking under transformation. The combination of tracking and discovering, localizing and describing transformations from videos can be very useful for many applications: for example, producing training annotations on recorded demonstrations for robots, analyzing compliance and processes on the factory floor from videos, understanding animal behavior and development from camera traps and so on. In these applications, understanding and tracking object transformations is critical, and real-time processing is not needed.

Potential for Amortized Efficiency: While the spatiotemporal partitioning is presented to be dependent on input object masks for clarity, our actual implementation (available in supplementary code) treats partition construction as a single preprocessing step that is shared across all objects of interest. This design allows the computational cost to be amortized when tracking multiple objects simultaneously. After preprocessing, TubeletGraph operates at speeds comparable to the base SAM2 model.

Bottleneck Analysis: The primary bottleneck is in constructing the spatiotemporal partition, which requires independent SAM2 tracking for each identified spatial region. This process is inherently parallelizable. Furthermore, we envision that future end-to-end systems can track multiple objects without incurring cost linear in the number of objects in the video. While important for practical deployment, we consider this optimization direction orthogonal to our core contributions of tracking and describing object transformations.

---

W2. Limited handling of non-proximal transformations

A launched firework is indeed an interesting example. We believe that this transformation does start with two proximal components (firework leaving the firework box), but due to low frame rate, the two components would appear to be non-proximal. We find these cases to be rare in day-to-day scenarios. Furthermore, the proximity constraint in TubeletGraph is soft, as we use the candidate masks produced by SAM2.1 to measure proximity. Therefore, as long as SAM2 can connect these non-proximal objects, our method will still succeed.

---

W3. The hyperparameter selection process appears ad-hoc

While we agree that a more principled approach to hyperparameter selection would improve the method's robustness, we found TubeletGraph's performance to be already very robust across varying hyperparameter values. To verify this, we conducted comprehensive grid searches over both spatial proximity and semantic consistency thresholds on M3-VOS and VSCOS datasets. The results demonstrate impressive stability, verifying that the hyperparameters are robust and generalize well across datasets without additional tuning efforts.

---

W4. Missing ablation studies for the impacts of different external model choices

To address this concern, we conduct ablation studies on key components and their impact on system performance.

W4.1 FCCLIP vs. DINOv2 Analysis We evaluated our method's robustness by replacing FCCLIP with DINOv2 while keeping all other components unchanged. As shown below, performance remains remarkably consistent across both feature extractors, demonstrating that our framework is robust to the choice of visual encoder without requiring hyperparameter adjustments.

W4.2 - VLM Robustness for Semantic Evaluation We further evaluate semantic correctness when GPT-4.1 is replaced with the open-source Qwen2.5-VL-7B-Instruct. We show comparable performance across both models, with Qwen marginally underperforming GPT4.1.

Due to rebuttal period constraints, we will extend this analysis to include CropFormer (replacing with Mask2Former) and SAM2.1 (replacing with Cutie or SAMURAI) in the camera-ready version for a comprehensive robustness evaluation.

We thank reviewer zLVN for their insightful comments and suggestions. We offer details to address the concerns below:

---

W1. Incremental progress

To clarify, our paper proposes the following contributions:

• We introduce a novel task of “Track Any State,” which requires simultaneously tracking objects through transformations while detecting and describing those transformations. As shown in Fig. 1 (bottom), no existing methods can both predict complete object tracks and at the same time provide spatiotemporal grounding and a semantic description of the transformation.
• We introduce a novel approach for recovering objects after transformations by leveraging a spatiotemporal partition of the input video.
• Our method tackles the novel task without requiring any labeled transformation data or training, making it broadly applicable across domains.

---

W2. Small empirical advantage for 2/4 datasets

We would like to point out that of the 2 datasets the reviewer is concerned about, DAVIS does not contain transforming objects. We include this benchmark to validate that our method doesn't overfit to transformation scenarios / degrade performance when objects do not undergo transformation. On the other datasets with transformations, our approach provides consistent significant improvements over SAM2.1.

In addition to a consistent, statistically significant improvement over the base SAM2.1 for tracking transforming objects, TubeletGraph does demonstrate an entirely new capability: simultaneous detecting and describing transformations. Here, we would like to note the strong performance in the temporal localization (Sec. 4.2) and semantic accuracy (shown below in Q3. How reliable is the GTP-4’s description?).

---

W3. Missing comparison with SAM2.1++

SAM2.1++(CVPR2025) was published after our submission deadline. We benchmarked it via its official repo below.

We see that TubeletGraph consistently outperforms SAM2.1++ on all transformation-focuses datasets: VOST (+3.3 J_tr), M3-VOS (+3.1 J_tr), and VSCOS (+6.2 J_tr), while maintaining similar performance on non-transformation dataset: DAVIS (85.6 vs 86.2 J)

---

W4. Object recovery depends on fixed thresholds

We found TubeletGraph's performance to be robust across varying threshold values. To this, we conducted comprehensive grid searches over both spatial proximity and semantic consistency thresholds on M3-VOS and VSCOS datasets. The results demonstrate impressive stability, verifying that the hyperparameters are robust and generalize well across datasets without additional tuning efforts.

---

W5. Limited qualitative / diagnostic analysis of failure modes

Qualitative Analysis: We showcase representative failure cases in the supplementary HTML page.

Representative failure cases in transformation descriptions

Video ID: 0315_paint_cream_4

• Error: The VLM misidentifies dropped paint cream as "meringue"
• Analysis: Both share similar visual appearance (white, creamy texture). With a plain background and limited contextual cues, the VLM cannot distinguish between two.

Video ID: 7359_fold_tape_measure

• Error: The tape measure case is misrecognized as a "smartphone," and the extended measuring tape is described as a "metal rod." Consequently, the transformation is described as "attach" rather than the correct "extend" or "pull out."
• Analysis: We believe that this failure stems from incomplete object views in the selected frames passed to the VLM, where hand occlusions prevent accurate object recognition and lead to cascading error for the action description.

Representative failure cases in tracking performance

Video ID: 0340_fry_omelet_1

• Error: A false positive omelet is incorporated into the state-graph at t=6s.
• Analysis: the flipping of a nearby irrelevant omelet causes a new tubelet to initiate, which was falsely incorporated to the original tracked batter due to high semantic and proximity similarity.

Video ID: 0357_make_pancakes_3

• Error: False negative regions are found during object transformation.
• Analysis: This is likely caused by a fast rotating occluder spatula altering the appearance of the transforming object rapidly. Once the transformation is complete, all associated regions are properly tracked.

Diagnostic Analysis: We observe a reduction in recall as we improve precision via the filterings (Tab. 1). This indicates that some positive objects are lost as we filter out unrelated instances.

We promise to add more failure cases and the above discussion in the camera ready.

---

W6. Missing references

We appreciate the references mentioned!

Indeed, the event segmentation discussed by Zacks and Swallow [A] does relate to the predicted state graph well. In fact, when evaluating temporal localization, each annotated transformation does fall near an event boundary, with an altered object state indicating a low-level goal of the activity being achieved. However, not all event boundaries will contain a state-altering transformation (two people playing catch can have well-defined action boundaries without any transformation to the ball). In our task of “Track Any State,” the goal is to detect and describe any object state transformations while maintaining consistent track of the target object.

While Voyager [B] focuses on LLM-driven embodied agent acquiring new skills in a simulated world engine without human intervention, its mechanism of prompting LLM for generating new tasks and programs is similar to our usage of prompting VLM to describe the detected transformation from before/after visual observations (Fig. 2(3)).

The benchmark by Wu et al. [C] focuses on bounding box tracking of transforming objects without any notation of detecting or describing the transformation. In our proposed “Track Any State,” the goal is both detecting and describing any object state transformations and maintaining accurate object tracks.

We will integrate these discussions in the camera-ready version.

[A] Zacks & Swallow. “Event Segmentation.” Current Directions in Psychological Science (2007).
[B] Wang et al. “Voyager: An open-ended embodied agent with large language models.” arXiv preprint arXiv:2305.16291 (2023).
[C] Wu et al. “Tracking Transforming Objects: A Benchmark.” arXiv preprint arXiv:2404.18143 (2024).

---

Q1. Can the system handle more complicated or slow changes?

Yes! Please refer to the video examples in the supplementary HTML page. We list some representative examples below.

• Video ID: 0315_paint_cream_4: dropping multiple droplets of paint cream.
• Video ID: 0269_press_pencil_1: pressing a pencil tip until it shatters
• Video ID: 0250_splash_liquid_2: dropping a disk into a bowl of water
• Video ID: 0369_cut_cheese_3: cutting into a cheese ball containing both solid and liquid components.
• Video ID: 0357_make_pancakes_3: Spreading pancake batter
• Video ID: 7359_fold_tape_measure: Extending and folding a metal tap measure

---

Q2. Is it a problem that fixed rules are used to find missing object parts?

Although a more elaborate system may improve performance, we believe that the fixed rules/thresholds allow a very robust system without need for dataset-specific tuning (shown above in W4. Object recovery depends on fixed thresholds), at the same time leveraging the modeling capabilities of the underlying models (e.g., FCCLIP for semantic similarity and SAM2’s candidate masks for proximity measure).

---

Q3. How reliable is the GTP-4’s description?

We found TubeletGraph to predict reliable action verbs (4.28/5) and resulting objects (4.38/5) as evaluated by human raters with very high reliability (ICC(2,k) > 0.9). For details regarding the experimental setup, please refer to our response to Reviewer jTZM, W1.

We thank reviewer jTZM for their insightful comments and suggestions. We offer details to address the concerns below:

---

W1. Missing semantic evaluation for predicted state graphs

To address this, we conducted additional experiments to evaluate the semantic accuracy of the predicted state graphs.

Experimental Setup: We first manually annotate captions for each object transformation used in our temporal localization evaluation (Section 4.2). Then, we consider any predicted transformations within 2 seconds of the GT localization as temporally correct candidates for evaluation. To measure semantic accuracy, we consider both the predicted action verb and resulting object state. We ask 3 human raters to evaluate the semantic similarity between predicted descriptions and ground truth annotations on a 1-5 scale.

Results

• We find high semantic correctness for both the predicted action verbs (average rating of 4.28/5 with 71.8% receiving perfect 5/5 scores) and resulting object states (average rating of 4.38/5 with 71.8% receiving 5/5 scores).
• The reliability of the human raters are confirmed by intraclass correlation coefficients (ICC) all above 0.9.
• This validates that the state graphs accurately capture the semantics of the object transformation.

Due to the constrained rebuttal period, we are not able to extend the label set further, but we promise to extend both the semantic and temporal localization evaluation to the entire VOST val in the camera-ready.

---

W2. High computational cost

We acknowledge the concern regarding computational cost and agree that the current speed limits real-time applications. However, we believe TubeletGraph’s unique capabilities provide significant value despite this limitation.

Value Beyond Real-Time Performance: TubeletGraph enables detecting and describing object state transformations beyond object tracking in videos. No existing methods perform both temporal localization and semantic description of the transformation in addition to state-of-the-art performance on object tracking under transformation. The combination of tracking and discovering, localizing and describing transformations from videos can be very useful for many applications: for example, producing training annotations on recorded demonstrations for robots, analyzing compliance and processes on the factory floor from videos, understanding animal behavior and development from camera traps and so on. In these applications, understanding and tracking object transformations is critical, and real-time processing is not needed.

Potential for Amortized Efficiency: While the spatiotemporal partitioning is presented to be dependent on input object masks for clarity, our actual implementation (available in supplementary code) treats partition construction as a single preprocessing step that is shared across all objects of interest. This design allows the computational cost to be amortized when tracking multiple objects simultaneously. After preprocessing, TubeletGraph operates at speeds comparable to the base SAM2 model.

Bottleneck Analysis and Potential Improvements: The primary bottleneck is in constructing the spatiotemporal partition, which requires independent SAM2 tracking for each identified spatial region. This process is inherently parallelizable. Furthermore, we envision that future end-to-end systems can track multiple objects without incurring cost linear in the number of objects in the video. While important for practical deployment, we consider this optimization direction orthogonal to our core contributions of tracking and describing object transformations.

---

W3. Missing analysis of external models and error propagation

To address this concern, we conduct ablation studies on key components and their impact on system performance.

W3.1 FCCLIP vs. DINOv2 Analysis We evaluated our method's robustness by replacing FCCLIP with DINOv2 while keeping all other components unchanged. As shown below, performance remains remarkably consistent across both feature extractors, demonstrating that our framework is robust to the choice of visual encoder without requiring hyperparameter adjustments.

W3.2 - VLM Robustness for Semantic Evaluation We further evaluate semantic correctness when GPT-4.1 is replaced with the open-source Qwen2.5-VL-7B-Instruct. We show comparable performance across both models, with Qwen marginally underperforming GPT4.1.

W3.3 - Error propagation After analyzing the failure cases, we identified that errors typically manifest as (1) false positive predictions by the base tracker and (2) minor reduction in tracking recall when applying semantic / proximal constraints (Tab. 1). False positive errors (1) can cause erroneous measures of semantic / proximal similarities, while reduction in tracking recall can reduce recall for temporal localization of object transformations.

Due to rebuttal period constraints, we will extend this analysis to include CropFormer (replacing with Mask2Former) and SAM2.1 (replacing with Cutie or SAMURAI) in the camera-ready version for a comprehensive robustness evaluation.

---

W4. No threshold tuning on other datasets (generalization is not shown)

We did not tune hyperparameters on other datasets since we observed TubeletGraph's performance to be robust across varying threshold values. To validate this observation and address the generalization concern, we conducted comprehensive grid searches over both spatial proximity and semantic consistency thresholds on both M3-VOS and VSCOS datasets.

Hyperparameter Robustness Analysis

• The grid search result shows stable performance across large parameter ranges, with maximum performance differences across a large set of parameter ranges being minimal (M3-VOS: 74.3 vs 72.8; VSCOS: 75.9 vs 75)
• The parameters tuned on the VOST train (sem_thrd=0.7, prox_thrd=0.3, found in center grid) perform competitively across all datasets.
• This verifies that the hyperparameter selection generalizes well across datasets without requiring dataset-specific tuning. This robustness further underlines the stability of our filtering mechanism without a need for precise threshold calibration.

---

Q1. Examples where the LLM's description of the transformation is incorrect or vague

Please refer to the supplementary HTML page for the following representative failure cases.

Video ID: 0315_paint_cream_4

• Error: The VLM misidentifies dropped paint cream as "meringue".
• Analysis: Both share similar visual appearance (white, creamy texture). With a plain background and limited contextual cues, the VLM cannot distinguish between two.

Video ID: 7359_fold_tape_measure

• Error: The tape measure case is misrecognized as a "smartphone," and the extended measuring tape is described as a "metal rod." Consequently, the transformation is described as "attach" rather than the correct "extend" or "pull out."
• Analysis: We believe that this failure stems from incomplete object views in the selected frames passed to the VLM, where hand occlusions prevent accurate object recognition and lead to cascading error for the action description.

We will provide additional failure case examples and analysis in the camera-ready version.

---

Q2. Applying TubeletGraph on top of SAM2.1 (ft) and quantifying upper-bound performance

Thank you for the suggestion. We evaluated TubeletGraph using the fine-tuned SAM2.1 (ft) model as the base tracker below.

We find that the integration of SAM2.1 (ft) with TubeletGraph (Ours+FT) shows modest improvements over the base TubeletGraph. However, the improvements are smaller than expected, given the strong standalone performance of SAM2.1 (ft).

We reason that is because TubeletGraph specifically addresses false-negative predictions by incorporating new candidate tracks lost due to object transformation. Since SAM2.1(ft) is fine-tuned on VOST to minimize these false negatives caused by transformation, the complementary benefits are naturally reduced.