STUPD: A Synthetic Dataset for Spatial and Temporal Relation Reasoning

Dear Reviewer, thanks for taking the time to review our paper.

1. " have not found details about which relations have been conducted experiments on. ImageNet-VidVRD defines 132 predicates. Some of them are "static" but not "dynamic" spatial relations. Also, the spatial relations are connected with verbs e.g., swim_behind, and fly_behind. It is not clear howto handle the different definitions of spatial relations during pre-training."

Thanks for pointing this out. We will add more details to the paper. In the ImageNet-VidVRD, we combine all similar relations (eg, behind, swim_behind, crawl_behing, fly_behind). "Swim_behind", for example, is just a subset of behind. As to how we handle static vs dynamic relations -- we convert all relations into "dynamic" representations, i.e., the same 3D model handles both dynamic and static relations. Static images are copied n times to create a 3D representation. Our class-wise experiments (in Table 2) suggest that the model can disambiguate between true dynamic 3D representations, and static 3D representations.

2. "There are some uncovered spatial prepositions among the 30 defined spatial relations. It seems that they have been collected but not evaluated. Since STUPD is a synthetic dataset that suffers from huge gaps with real-world images. It is important to verify its effectiveness on real-world tasks". Unfortunately, there are no real-world visual relation datasets, to our knowledge, that cover the variety of relations that we have collected. This is, thus, a reflection for other researchers to expand their work to cover these relations as well. For current datasets that cover the limited number of spatial relations, it makes sense to only pretrain on the subset of STUPD that overlaps with the datasets. This helps us perform systematic ablations about the specific knowledge of individual relational meanings imparted by STUPD. As to whether or not the uncovered relations will be useful for real-life settings, our experiments provide strong arguments in that direction. We systematically showed that STUPD-pretrained models perform better on real-world datasets containing relations as we define in our paper. Since, the data designing and generation process is consistent across all prepositions, it is logical to deduce that overall, other prepositional relations in STUPD would also be useful to real-world settings.

3. "I wonder about the necessity of prosing a new task of temporal relations." Actually your argument is correct. An oracle model with time-stamp based inputs is not really useful, since one can differentiate between many relations with simple rules. Based on another reviewers comment, we introduced another experiment (involving a VQA dataset), that shows how pretraining on temporal STUPD can also help in real-world settings. In this model, we evaluate whether models can learn temporal relations by analysing the videos, and understanding the nature of the individual events. We included many changes in the paper, all highlighted in the color blue.

Dear Reviewer, thanks for taking the time to read our paper in depth and provide critical comments. Your reviews helped us improve the quality of the paper significantly. We try to answer your concerns below.

1. "Lack of use of the STUPD dataset as a temporal reasoning pretraining set.". You are right, and after several discussions, we decided to proceed with a pretraining experiment for Temporal-STUPD as well, which involves a VQA dataset. While most VQA datasets either focus on spatial relations only, or do not provide structured temporal relation annotations, we observe that the NeXT-QA dataset [1] is suitable for our needs. Another candidate -- the TEMPO dataset [2] would have been useful, but unfortunately the dataset has not been uploaded publically in its entirety by the authors. Please see Section 4.2 of the revised paper. Similar to Spatial-STUPD, we observe that Temporal-STUPD too helps in improving real world temporal relation reasoning significantly.

2. "Reason for splitting the temporal relationship into ten categories [is unclear]". The main goal of STUPD is to not just provide synthetic visual representations of common prepositional relations, but to also help models disambiguate between the different "senses" of a word, or in other words, the different meanings of the same word in different context. For example, "along" in Spatial-STUPD has two meanings in different contexts. Similarly in Temporal-STUPD, although "before" is a subset of "by", they have different subtle meanings. For example, "before" can be used to depict a relation between two events (event A before event B), or an event and a time point (Event A before Time B), but "by" is explicitly used to depict a relation between an event and a time point (Event A by Time B). (You also mentioned "Also, I don't think "by" (which implies a deadline) is used to describe the timing of two events in general". This is true, and has already been factored in the paper. See Figure 2 in the paper).

3. " Clarity. In the #1 callout in Sec 3.4.1, the author pointed out that 'track' and 'tunnel' have fewer relationships than other object types.". This design is quite very intentional, and does not arise out of limitations in UNITY. As we mentioned in the paper, we follow a structured object-pair selection process to generate the dataset. This involves filtering out object-pairs that are unlikely to occur in the real world, and also a size constraint (both objects in the pair should have similar scales of size, otherwise the resulting images would not be able to capture both objects without affecting the visibility of at least on the objects in the object pair). Due to this, tunnels and track (being large scale objects) are paired with only other larger objects (like vehicles, furniture, buildings, etc). The other reason for the lesser frequency of tunnel/track is that in the spatial relation triplet (Object A, relation, Object B), tunnels and tracks are semantically restricted to Object B. It is unlikely to see tunnel and track being the subject of a sentence (eg. a track is above a car is not only logically inconsistent, but also semantically incorrect). This is why, overall, we see lesser occurrences of these two types of objects.

[Minor corrections] 4. citation format. Thanks for pointing this out. We have rectified this now. 5. "The author frequently used "<" and ">" without any escape character." This is corrected too. 6. "2nd line of 2nd para of Intro. I cannot understand what pre is for. ". This was a wrong citation, and has now been corrected. 7. formating of large numbers with Commas. Corrected.

[1] Xiao, J., Shang, X., Yao, A., & Chua, T. S. (2021). Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9777-9786).

[2] Hendricks, L. A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., & Russell, B. (2018). Localizing moments in video with temporal language. arXiv preprint arXiv:1809.01337.

Dear Reviewer, thanks for your comment. Understandably, your concerns are valid. We had this concern right from the beginning, and took many steps to ensure that a synthetic dataset is useful to real world applications as well.

1. "The synthetic data may not fully capture the complexity of real-world scenarios." We ensured that the selection of objects, and the nature of interaction was as realistic as possible. Not only did we use textured real-world-like objects, but restricted object pairs to only those that are likely to be seen in the real world. Also the physics engine used ensures that the interaction is natural.

2. "The paper could benefit from a more extensive validation of the dataset's efficacy across a broader range of models and tasks." Agreed. We realize that there could be more experiments for the temporal part of STUPD. So we will include some results along those lines as well.

3. "How well do models trained on STUPD perform when applied to real-world data, considering the dataset is synthetic?" We demonstrated through the pretraining experiments (Table 3 and 4), that when models are first pretrained on STUPD, and then fine tuned on real-world datasets, the model performance is. consistently better.

4. "How does STUPD handle ambiguous or context-dependent prepositions where the spatial or temporal relationship might not be clear-cut?" Different context dependent prepositions have distinct labels and videos corresponding to them. For example, the word "against" has two meanings, and different corresponding videos. Similarly, there are two different classes corresponding to "into". Within certain constraints, we cover all prepositional relations in our dataset. Maybe some less-frequently words are not covered, but their meaning is covered by one or the other of the classes in this dataset. There are 40 classes in total, and the full list can be seen in Appendix.

5. "What measures are in place to ensure that the synthetic data in STUPD is diverse and representative of real-world scenarios?" We started out by selecting a diverse range of objects from various supercategories (like vehicles, large grounded objects, small objects, larger objects, containers). Overall, the diversity of objects as well as the absolute number of objects is comparable to SpatialSense (a real world dataset). These synthetic objects have real-world-like textures. We also filter out object pairs that are unlikely to be seen in the real world (such as building on top of a car). Backgrounds are also selected so as to bring realness. Finally, a physics engine ensures that the interactions are real-world-like.