Where is the Invisible: Spatial-Temporal Reasoning with Object Permanence

Q4: Some detailed designs are not explained clearly...

A4: We are sorry that we did not describe the details exhaustively and caused your misunderstanding. We have revised the manuscript to explain more details. For convenience, we introduce the details in the following:

Object Detection: For the experiments on the CATER and LA-CATER datasets, due to the limited types of objects included in the dataset, the vision module can be implemented by an object detector. For a fair comparison, we use the same object detector as previous works (OPNet[1], AAPA[2], and RAM[3]), a Faster RCNN network pre-trained on the COCO dataset, which is introduced in Section 4.2 of the paper. For the experiments in real scenes, we apply AutoMatch[4] as the object detector. During tracking, the search area in the next frame depends on the estimated target location (detection position or predicted position) in the current frame. This rule is widely used in visual tracking models. In this case, the adjustment of the search area will affect the performance of the tracker once it disappears. Hence, in our framework, once the confidence of the detection result is lower than a threshold of $\lambda_{disappear}$, we will assume the target has disappeared and predict the trajectory of the invisible target to guide the adjustment of the search area synchronously. In the same way, when the confidence of an object that originally disappeared is greater than $\lambda_{reappear}$, we will think that the object appears again. The proposed method focuses on predicting the spatial and motion relationships of objects in the invisible state, rather than vision-based object tracking.

Correction Module: Thanks for pointing this out. Since the frame rates of the LA-CATER synthetic dataset and the iVOT real-world dataset used are both 30fps, the scale of objects in adjacent frames changed slightly in most cases. If the scale changes dramatically, the correction module may result in unreasonable modifications. It will be our future work to extend our framework on videos with violent shaking.

Q5: One of the main contributions of this paper is to propose a new dataset, however, there are very few descriptions of this new dataset. Besides, there is also a lack of experimental results of SOTA methods on the proposed dataset iVOT (Table 3).

A5: We appreciate your suggestions. We have added more details and results about the new dataset. To be specific, the proposed iVOT(invisible Object Tracking) dataset is collected by Intel Realsense D435i which has an RGB frame resolution of 1920*1080, a depth output resolution of 1280*720, and a frame rate of 30. The data are recorded in indoor rooms (e.g., bedroom and living room), including diverse human-object interaction activities, where the objects frequently become invisible in the video. These cases are normal in our daily activities but are challenging to the state-of-the-art methods. For example, a man picks up a toy into a box and moves the box to another place, where the model needs to reason the object-object relation to localize the target object. More details on this dataset are elaborated in Appendix A.5. We also add more results of SOTA methods and visualizations in Appendix A.6.

• Comparison of SOTA methods testing on the iVOT dataset using mean IOU.

Q6: The paper is a bit hard to read...

A6: Thanks for your valuable and helpful suggestions. We have corrected relevant ambiguities or inconsistencies in the revision. We highlight the main update in blue.

Thanks for your helpful and constructive suggestions. Below we answer the questions in detail.

Q1: The different modules in the proposed framework don’t look compact, but independent. The framework integrates many separate models to address separate problems, like an object detector, a human pose estimation, the correction module, and a diffusion-model-based trajectory predictor. These different models are also trained separately, but not in an end-to-end manner. Compared with SOTA methods, the results in Table 2 are also not evident enough to show the advantages brought by this kind of complex design.

A1: We argue that the overall framework is well-organized and comprehensive, rather than independent. All the modules we introduced are essential for the reasoning of object permanence. The object detector localizes the visible objects, and the human pose estimation is to recognize the human-object interaction for identifying the change in the object-object relationship (as most object states are changed by human actions in the real world). To localize the invisible object, the trajectory prediction module proposes multiple possible trajectories conditioned on the reasoned spatial relationship. The correction module checks the reasonableness of the proposed trajectory based on the state of the visible objects. Each module is interdependent and indispensable for localizing the invisible objects, as shown in the ablation study.

Compared to the end-to-end methods, our method has the following advantages:

1. Generalizability. The pre-trained model can provide generalizable state representation to make our framework easily transferable to unseen datasets. We have reported the cross-domain generalization of our method compared with the baselines, shown in the following Table. The models are trained on LA-CATER and tested on iVOT. Our approach demonstrates superior generalization capabilities compared to OPNet.

2. Modularity. Each module (e.g., object detection, action recognition, trajectory prediction) can be easily replaced by state-of-the-art methods, thus our model can benefit from the advances of each component.
3. Interpretability. Our framework consists of modular components that can be individually analyzed and evaluated. For example, we can visualize the relationship graph and the forecasted trajectory produced by each module, which can help us identify the potential weaknesses and limitations of our approach.

Q2: Besides, the framework seems to bring many hyperparameters, I think it's necessary to explain these hyperparameter settings, and also perform corresponding experiments to show the robustness of the proposed framework on different hyperparameter settings.

Thanks for your suggestions. We have provided the details about the hyperparameter in the Appendix. For convenience, we listed the main hyperparameters used in our method and conducted experiments to further analyze the effect of each hyperparameter in the LA-CATER. In the ablation experiment, we fixed other hyperparameters and varied only one hyperparameter to test the impact of different hyperparameters on performance stability. As is shown in the results, our framework can be robust to the change of hyperparameters. We have added this analysis in Section 4.3 and Table 2 of the revised manuscript.

• Experimental results of our method on the LA-CATER dataset with different hyperparameters using mean IoU. We highlight the default value in bold.

We wish to thank the reviewer for this helpful and comprehensive referee report, and we are glad to see your positive assessment of our methodology and presentation. Below we answer the questions and explain how the revised manuscript accommodates this detailed and helpful feedback.

Q1. Looking at a high level, the current method mimics the classical Kalman and Particle filter very closely...

We applied the traditional Kalman filter method to conduct experiments both on the LA-CATER and iVOT datasets, and the experimental results showed that the traditional filter method cannot effectively understand the spatial temporal relationship under uncertain noise distribution and nonlinear system. That is the Kalman filter only predicts the short-term trajectories, and can not conditioned on the object-object relationship. However, the objects in containment may disappear in the long term.

• Comparison of the traditional Kalman filter method and our proposed method on the LA-CATER dataset and the iVOT dataset using mean IOU.

Q2. Authors demonstrate results on two real-world benchmarks but I believe some more experiments are necessary to properly understand the contributions.

We agree that the evaluation of more real-world datasets is necessary. However, we notice that the object-object relationship in most existing real-world datasets is too simple to evaluate the object permanence reasoning. The invisible objects usually are just occluded by other objects or out of view, hardly contained by other objects. As a result, the invisible objects usually are shortly occluded by other objects, and the reasoning is unnecessary in most cases. However, in this paper, we focus on the ability to reason the localization of invisible objects, partially in some complex relationships, e.g., co-occurring containments and occlusions. Hence, this motivated us to collect new datasets in our daily activities that contain complex transitions of the object relationship. We argue that it will be our future work to collect more diverse videos with complex object relationships to evaluate the object permanence, rather than on those existing video tracking datasets, as the focus of the two domains are different.

Q3. A few design decisions haven't been ablated to understand their significance. Why did the authors choose to go with diffusion models for object tracking? Why not go with the well tested trackers like SORT[2] or DeepSORT[3]? Or some recent state of the art MOT trackers [4]? I think this experiment is necessary to justify the use of diffusion models.

The main purpose of using the diffusion model is to predict the trajectories of invisible objects under certain spatial constraints rather than serving as a tracker itself. Under high uncertainty, the possible trajectories of the invisible target can be viewed as a noisy Gaussian distribution that represents a blurred area controlled by the container trajectory. As the uncertainty decreases, the distribution gradually approximates the ground truth distribution for generating the correct trajectory. Therefore, we regard the trajectory prediction task as a condition-based generative task and apply the diffusion model. The relationship experiments in Section 4.5 demonstrate the effectiveness of diffusion models for trajectory prediction.

The MOT trackers also apply a similar tracking-by-detection framework. They focus on solving the data association problem for multi-object identification, where the objects may be of similar appearance and will shortly occluded by others. In this case, most MOT trackers assume the object keeps the linear dynamic (they usually track multiple pedestrians), and only apply some simple models to predict the short-term trajectory when the object is occluded by another. However, such short-term occlusion is a simple case in the study of object permanence. We find that the trajectory of the object highly depends on the relationship to other objects. If the object is only occluded by others, it will keep its own movements. In contrast, if it is contained in some object, its trajectory will depend on the transition of the container. That is why we need to build a relation-conditioned trajectory prediction for tracking invisible objects. The results of the Kalman Filter-based tracker also show that the simple predictor can not handle the complex relationship and long-term invisibility.

Q4. Not a major issue but make sure the tables and figures in the paper appear at the top of the page.

Thanks to the reviewer for valuable and helpful suggestions. We have revised the manuscript to make the Tables and Figures appear at the top of the page.

No reply from the authors. Keeping my original rating for "5: marginally below the acceptance threshold".

We sincerely apologize for our delayed reply. We hope you will kindly reconsider your rating after reading our response. We appreciate your patience and understanding. We thank you for your helpful and comprehensive review. We have carefully addressed your questions and incorporated your feedback into the revised manuscript. The followings are the detailed response:

Q1: No qualitative results...

A1: We agree with your comment that qualitative results are important to demonstrate the effectiveness of our method. We have added representative sequences on the iVOT datasets in the manuscript, as shown in Appendix A.6. These sequences compare the qualitative results of different methods and illustrate the advantages of our approach.

Q2: Evaluation on the CATER dataset...

A2: We apologize for the confusion caused by our argument. We have evaluated our model on both the LA-CATER and the CATER datasets. Due to space limitations, we reported the results on the CATER dataset in Appendix A.3. We have clarified this point in the manuscript. In summary, our method outperforms most of the baselines in the Top 1 Accuracy and achieves comparable results to [1] (slightly lower in the Top 1 Accuracy but higher in the Top 5 Accuracy).

Q3: Effect of number of objects on performance/generalization...

A3: We believe that this problem can be solved to a certain extent by increasing the size of the candidate relationship sets ($K$) or through appropriate pruning methods. In fact, in the LA-CATER and iVOT datasets of our experiments, the maximum number of objects is $15$ (more details are shown in Appendix A.2 of our paper), and on this scale of numbers, our proposed method shows satisfying performance. Finding a way to efficiently establish spatio-temporal relationships in crowded scenes is one of our future research directions.

We also agree that the evaluation of more real-world datasets is necessary. However, we observe that most existing real-world datasets have simple object-object relationships that are not suitable for evaluating object permanence reasoning. As a result, the invisible objects are usually briefly occluded by other objects, and the relation reasoning is trivial in most cases. In contrast, in this paper, we focus on the ability to reason about the localization of invisible objects, especially in some complex relationships, such as co-occurring containments and occlusions. Therefore, we collected new datasets from our daily activities that contain complex transitions of object relationships. We argue that it is more meaningful to evaluate the object permanence on these new datasets, rather than on the existing video tracking datasets, as the two domains have different focuses.

Q4: More descriptive method figures...

A4: We appreciate your valuable suggestions. We have revised our manuscript to provide more details about each module of our method. Please refer to Appendix A.1 (Fig.4, Fig.5, Fig.6) for the detailed illustrations and explanations of our method.

[1] LEARNING WHAT AND WHERE: DISENTANGLING LOCATION AND IDENTITY TRACKING WITHOUT SUPERVISION, ICLR 2023.

Q5: Despite a relatively straightforward main idea of estimating occlusion /containment status based on past trajectories, there are a lot of twists and tweaks involved...

Compared to the end-to-end methods, our method has the following advantages:

1. Generalizability. The pre-trained model can provide generalizable state representation to make our framework easily transferable to unseen datasets. We have reported the cross-domain generalization of our method compared with the baselines. The models are trained on LA-CATER and tested on iVOT. Our approach demonstrates superior generalization capabilities compared to OPNet.
2. Modularity. Each module (e.g., object detection, action recognition, trajectory prediction) can be easily replaced by state-of-the-art methods, thus our model can benefit from the advances of each component.
3. Interpretability. Our framework consists of modular components that can be individually analyzed and evaluated. For example, we can visualize the relationship graph and the forecasted trajectory produced by each module, which can help us identify the potential weaknesses and limitations of our approach.

To evaluate the generalization ability of our method to different datasets, we performed cross-dataset experiments, where we trained our model on the synthetic LA-CATER dataset and tested it on the real-world iVOT dataset. In this experiment, we use OPNet for comparison. The AutoMatch model is a part of the vision module for localizing the visible objects. Results show that our proposed method has stronger generalization ability and performance than OPNet.

• Comparison of the cross-domain generalization using mean mIoU. We train the models on the LA-CATER dataset and evaluate the performance on the iVOT dataset. We also report the results of OPNet as a reference.

Q6: The presentation needs to be improved.

Thanks for your valuable and helpful suggestions. We have revised relevant ambiguities or inconsistencies in the paper.

[1]. Zhang, Zhipeng, et al. "Learn to match: Automatic matching network design for visual tracking." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2]. Shamsian, Aviv, et al. "Learning object permanence from video." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16. Springer International Publishing, 2020.

Thanks for the helpful and comprehensive review, and we are glad to see your positive assessment of our methodology ("Adapt a diffusion model for trajectory generation looks interesting") and experiments. Below we answer the questions and explain how the revised manuscript accommodates this detailed and helpful feedback.

Q1: Many important details are not well explained or are missing.

We apologize for the lack of clarity and detail in our original manuscript. We have improved our manuscript to provide more details about our method. In particular, we have explained how the hand positions are used in the Spatial Relation Predictor (Section 3.4). Empirically, we regard the hand as a special object that never be contained by other objects in our framework. If we treat the hand as an ordinary object, it may cause ambiguity between containments and occlusions.

Q2: When generating spatial relationship graphs, do you distinguish occlusion and containment? If not, why?

When generating a spatial relation graph, we distinguish three types of spatial relationships: occlusion, direct containment, and no direct relationship. We use different edge categories to represent these relationships.

Q3: when generating the graphs, why do you only use object trajectories without identity information? The object characteristic should affect occlusion and containment, in my opinion.

The visual module in our framework has already obtained the trajectories of visible objects by ID matching (using the same Faster RCNN network as OPNet and RAM on the LA-CATER dataset, and the AutoMatch tracker on real-world datasets). In this paper, the relation graphs focus on the relationship between invisible objects and visible objects. However, there is no identity information for invisible objects, so our method only uses historical trajectories without identity information when generating spatial relation graphs. The main goal of our work is to infer the trajectories of invisible objects by constructing the spatial relations of both visible and invisible objects, rather than using visual features to track visible objects.

Q4: The method needs to "enumerate all possible spatial relations in the first frame and form the candidates PG1." This may cause some problems when there are too many objects.

We agree that the number of candidate relationship graphs may grow exponentially, so we only keep the most likely $K$ spatial relation graphs at each time step, as described in the Evaluation Module in Section 3.5. In practice, our method achieves an average processing speed of $100ms$ per frame when the size of the candidate set K=16. We have also tested the cases where the size of candidate sets is larger or smaller, and the processing speed of our method is about $30ms$ per frame (K=1) and $250ms$ per frame (K=64). We have updated the manuscript accordingly.