Can Transformers Capture Spatial Relations between Objects?

Thank you very much for the insightful suggestions and supportive comments. We hope the following response will address your concerns.

Q1: It is unclear if the newer annotation provides a better evaluation or a simpler task. It is possible that the new annotations are more strongly correlated with the relative locations in the image, which would introduce a strong prior.

Answer: More precise definition does indeed result in an easier task. Imprecise definitions and the resulting inconsistent annotations lead to "irreducible errors" and therefore worse performance metrics. More importantly, the resulting models are less useful, because it is not clear what meaning to assign to the models if the task is specified in this imprecise/inconsistent way.

Q2: The newer annotation scheme reduces the ambiguity but it also makes the relationships less dependent on context. For example, the definition of “in front of” in the paper is a much easier one than the typical usage which often also requires reasoning about the subject and object's relative orientation.

Answer: Thanks for this example. In our opinion, this is only true for objects with "a canonical front". Not all object categories have such properties -- most don't. Furthermore, note that this involves reasoning through semantics. We do believe there is potential value for such a task, but it is worthwhile to decompose the geometric and semantic components of the task. i.e., first use our geometric notion of spatial relationships, then identify the semantic spatial orientations of the objects, and then combine these inferences into a "semantic spatial relationship" such as "human A is in front of human B".

Q3: It is unclear how the method works without Pair Interaction, Context Aggregation, or Feature Extraction.

Answer: In our model, Pair Interaction is implemented by the attention mechanism between subject and object query tokens within the transformer. We ablate it by applying an attention mask on the attention weight between these two tokens. For Context Aggregation, similarly, we employ an attention mask on the attention weights between the query tokens and the image patch tokens. For Feature Extraction, we remove the ViT Encoder and relied solely on the patch embeddings of the image patches to predict spatial relations. We will describe the ablation experiment setup more clearly in our revision.

Q4: Provide the CNN model with some positional embeddings, e.g., through CoordConv.

Answer: Thanks for your valuable suggestion! Following your advice, we incorporate positional embeddings into the input tensor of the first layer of ResNet in both DRNet and RUNet. The results, as illustrated in the table below, indicating that the CNN-based models (DRNet and RUNet) did not benefit significantly from the addition of positional embeddings. Their performance remains inferior compared to RelatiViT. This outcome reinforces our hypothesis that RelatiViT’s superior performance is largely due to its enhanced ability to capture spatial relationships through attention mechanisms, as opposed to the sliding convolution used in traditional CNNs.

Q5: More analysis is required to make the statement that the prior methods tend to make predictions mainly according to bounding box coordinates.

Answer: Thank you very much for this suggestion. To verify whether the models predominantly rely on bounding box coordinates over image content, we conducted an intervention experiment by replacing the input images with random ones during testing. As shown in the following table, we report the prediction flipping ratio, which measures the frequency of predictions changing from True to False, or vice versa, across all samples. A higher flipping ratio suggests greater reliance on image content, while a lower ratio indicates a tendency to base predictions on bounding box coordinates. We find that DRNet and RUNet exhibit significantly lower flipping ratios than RelatiViT, indicating that RelatiViT places more emphasis on visual cues to when predicting spatial relations, contrary to the baselines focusing mainly on bounding box coordinates.

Q6: Typo in loss function.

Answer: Thank you very much for pointing this out. We will correct it in our revision.

Once again, we are grateful for your precious time, insightful suggestions, and supportive comments. We will revise our paper according to your suggestions.

Thank you for engaging with all my comments and points. I respond below to each of those points. Overall, the points raised in Q1/2/6 are all addressed. I would appreciate the authors response regarding Q 3,4, and 5.

Q1/Q2: The point made here is the newer annotations are more precise, and while they may make the task easier, this is primarily by resolving ambiguity (A1) and relying more on geometric rather than semantic relationships (A2). Overall, I see your point and mostly agree with it. One extra point I will add is that this is not just geometric, it's geometric relationships defined in a primarily ego-centric frame of reference. This changes things a bit as you can have well-defined relationships that do not depend on the observer, and such relationships would have much weaker relationships to the bounding box locations as they would be less view-dependent. One suggestion, in the spirit of your work, would be to revise the naming in Table 7 as many of those relationships are defined in an object-centric frame of reference; eg, in-front-of is currently defined with respect to the object, however, the relationship is a function of both the viewer and the object and is completely independent of the object's pose, just its location. Anyway, I think any remaining disagreements are more subjective, as a result, I think those points have been addressed.

Q3: Thanks for the clarification. For pair interaction, does this mean that your attention matrix simply becomes a block matrix (with 4 blocks, 2 on the diagonal and 2 off the diagonal), where off-diagonal blocks are set to 0? if so, isn't that just equivalent to self attention with shared weights followed by concatenation?

Q4: I appreciate the additional experiments, however, I find it difficult to make fair comparisons as both DRNet and RUNet have specialized architectures and hence, adding CoordConv to them does not provide a clear comparison to your method. In my review, I suggested the impact of positional embeddings specifically on the results in Table 3 as this, to me, is the key results table in the paper as it very nicely tests the claims of the paper. The question I have pertains to adding positional embeddings before the RoI align in Figure 3b. The reason I think this is an important experiment is because it provides the transformer following the CNN encoder with positional embeddings, something that it natively gets in the ViT encoders. Furthermore, by comparing the model to the adapted CNN-Transformer Figure 3b, we can control for other effects such as the architectural choices of previous approaches potentially being counterproductive.

Q5: I appreciate going through the extra work. Overall, I really liked the experimental design of this experiment as it tests for sensitivity in a very nice way. However, my critiques in the above point hold. DRNet and RUNet do not provide a clear comparison to RelatiViT that addresses the main hypothesis/question of the paper as stated by the title and elaborated on in section 4.

Thanks for your valuable feedback. We hope the following response will address your concerns.

Q1: The applicability of the proposed method is limited because the spatial relations do not include the human-object and multi-object relations.

Answer: Human-object interaction and multi-object relations are indeed promising topics to investigate. However, we find the existing methods struggle to accurately resolve even the fundamental spatial relations addressed in our paper. In this paper, we aim to investigate this spatial relation prediction task to show the importance and challenges of this under-explored field. Our intention is to provide a foundational framework that can serve as a springboard for tackling more complex scenarios in future research.

As for the applicability, our method can be easily applied to the robot systems to query for spatial relations. This capability can be further leveraged for advanced applications such as PDDL-based planning or unsupervised reward learning, which will boost the performance of robot learning.

Q2: This paper would benefit from more analysis of the superior performance of RelatiViT and a thorough study on each proposed component.

Answer: Thanks for your kind suggestion and for recognizing the good performance of our method. As also mentioned by Reviewer XTRZ and Reviewer ubwR, we have provided extensive experiments to analyze the success of RelatiViT in Section 5.5, Appendix A5, A7 and A8. In Section 5.5, we provide an in-depth ablation study, revealing that both context aggregation and good feature extraction are key contributors to the performance. The attention maps in Section 5.5 and Appendix A8 demonstrate that RelatiViT attends to appropriate regions to aggregate the context information for accurate spatial relation prediction. And the comparisons on each relation category in Section 5.5 and Appendix A5 show that RelaTiViT significantly surpasses baselines (especially bbox-only) in categories where visual information is critical. This indicates that RelatiViT successfully extracts the visual representations containing rich spatial relation information. Additionally, the prediction visualizations in Appendix A7 showcase that RelatiViT predicts the relations based on the visual information, e.g., object contact, occlusion, depth and background context, etc. This capability fundamentally contributes to its superiority over bbox-only baselines.

Q3: "RelatiViT" does not consistently outperform the "bbox-only" baseline across various categories of spatial relations.

Answer: In Table 6, our results are either comparable or outperform the bbox-only baselines across most categories. Moreover, in Table 10, RelatiViT performs better than bbox-only on 29 out of 30 predicate categories, which is a significant improvement.

It is essential to recognize that while bbox-only models may achieve comparable accuracy in certain scenarios, such blind models, which do not leverage image inputs, possess inherent limits in further improvements. In contrast, our method leverages both visual data and advanced attention mechanisms, marking a notable advancement in predicting spatial relations. For the first time, our work demonstrates that vision-based methods can substantially exceed the bbox-only baseline. This achievement highlights the pivotal role of visual cues and relation modeling in this task. We believe that future research building upon our methodology will achieve even greater accuracy, far surpassing simpler baselines.

Q4: The reasons why "RelatiViT" exhibits superior performance in "left" but worse in “right”.

Answer: There might be some misunderstanding. According to Table 6, the performance of RelatiViT is comparable with bbox-only on “left” and “right”. However, it's important to highlight that RelatiViT demonstrates significantly superior performance on more complex spatial relations such as "behind", "in", "under", and "front". These particular relations inherently require a more nuanced understanding of visual information.

Thanks again for your precious time. Please let us know if there are any additional questions.

Thanks for your valuable suggestions. We hope the following response will address your concerns.

Q1: Quantitative bias analysis of linguistic biases.

Answer: To demonstrate quantitatively how linguistic bias affects annotation quality, we analyze the top 20 subject-predicate-object triplets with the most significant change in labels after our relabeling process in the SpatialSense dataset. We also provide the issue types that make the original annotation incorrect, including object-centric, polysemous words, and idiomatic expressions, i.e., the three types of errors as discussed in Section 3.2. Our findings reveal that idiomatic expressions account for 20% of the incorrectly labeled triplets, polysemous words contribute to 10% of these errors, and the remaining 70% is mixing up object-centric and viewer-centric relations. The percentages of idiomatic expressions and polysemous words highlight the typical failure cases caused by linguistic bias in the dataset.

Q2: Discussion about the statistics bias of spatial relations.

Answer: Thanks for this insightful question. The original SpatialSense dataset is collected with adversarial crowdsourcing strategy. Specifically, the authors trained a simple model to predict spatial relations mainly based on the statistics, and then they asked crowdsourcing annotators to only label the samples that cannot be handled by this model. In this process, the statistics bias has been much alleviated. Our SpatialSense+ retains these samples and predicate categories from the original dataset. Consequently, we inherit this advantage of mitigating statistical bias.

Q3: Compare with more advanced transformer-based models.

Answer: Thank you for your insightful suggestions. We apologize for missing references such as SGTR [1] and RelTR [2]. Firstly, we want to clarify that these works are in different settings from ours: SGTR and RelTR aim to generate scene graphs and include an end-to-end detection process, but in our setting the detection is not required. Recognizing the distinct settings of our work, we make much effort to adapt RelTR to align with our experimental setting. This involved maintaining the same ResNet backbone as RelTR, encoding bounding box and object category information into learnable entity tokens, and modifying the relation prediction head to mirror the K-way binary classification head used in our models.

Upon implementing these changes, we observed an average accuracy of 57.91% with RelTR, which is markedly lower than anticipated. We attribute this underperformance to two primary factors:

(1) RelTR relies on the features from only the last layer of ResNet, which we believe results in a significant loss of spatial information.

(2) RelTR leverages learnable tokens to establish correspondence between the queries and the feature in region of interests, and then aggregate the information. This is difficult to achieve by the supervision of a small training set. Differently, our model utilizing large-scale pretrained transformer features and design a mask image based query localization strategy, which is able to establish the correspondence with fewshot training samples.

This experiment further highlights the capability to capture spatial relationships of our model. We will definitely include these references in our revision.

[1] Li R, Zhang S, He X. Sgtr: End-to-end scene graph generation with transformer[C]//proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 19486-19496.

[2] Cong Y, Yang M Y, Rosenhahn B. Reltr: Relation transformer for scene graph generation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

We appreciate your precious time in reviewing our paper. Please let us know if you have additional questions.

Dear Reviewer,

Thanks again for your precious time and insightful suggestions. We hope we have addressed your concerns comprehensively. As the deadline is approaching, with less than 8 hours remaining, we kindly request that you let us know if there are any further questions or clarifications needed.

Best regards,

Authors

Thanks for the authors' responses. I think my concerns have been addressed. I also found the discussions of hAXs.1, ubwR.3, ubwR.5, Jenr.2, Jenr.5 valuable. Thus, I would like to raise my score to 6 and would not object to accepting this paper.

Thanks for your valuable suggestions. We hope the following response will address your concerns.

Q1: The proposed transformer-based method does not bring a lot of new insights because the superiority of transformer is commonly admitted.

Answer: Firstly, we want to restate that our argument in this paper is not only transformer is better than CNN, but also transformer architectures are not always optimal but need special design and correct visual prompts to read out the relation information, as we compared and discussed in Section 5.3. Besides, transformers actually have limited improvements on common tasks, e.g., image classification (top1 acc: ImageNet ResNet101 80.67 V.S. ViT-Base 80.73) and detection (COCO AP: Faster RCNN 42.0 V.S. DETR 43.3). In contrast, our transformer architecture shows significant performance boosting on the spatial relation task, as also mentioned by Reviewer XTRZ, MUvq and ubwR. We believe the architectural design principles will provide insight for the community.

Q2: There are natural ambiguities in the task of spatial relationships. So the claim of "a precise and unambiguous definition" seems a bit exaggerated.

Answer: A precise definition of the spatial relationship is possible with complete knowledge of the scene. This is different from whether a 2D image of the scene always presents this information unambiguously. This is also true for other types of visual properties that we like to infer, such as object shape: humans may not always be able to infer full 3D shapes of objects from an image, but the 3D shape is of course well-defined. Therefore, we provide precise and unambiguous definitions according to the geometries to annotate the dataset. As for the capability of vision model to solve this challenging task based on 2D images where there might existing ambiguity, we think it can be solved by large-scale pretraining and special architecture design. In this paper, we demonstrate this potential solution by our ReltiViT and IBOT pretrained vision transformer. We think this is a good starting point.

Q3: The definitions provided by the authors include subjective (which seems naturally inevitable for this task) appreciation.

Answer: Thank you for raising this point. It is indeed more precise to separate an "in-principle definition" from an "operationalizable definition" that an annotator can use to provide consistent annotations. We will clarify that these definitions are intended to focus human labeling efforts on unambiguously true predicates. In our annotation process, a sample was annotated by several annotators and the final label was voted among these multiple annotations, which reduces this problem.

(This response is continued in the next comment block.)

Q4: The contribution to dataset is limited because the author reuses the same predicates as the previous dataset.

Answer: We would like to clarify that the main problem with the original SpatialSense dataset for studying spatial relationship prediction is not that it lacks some predicates, or that it has an uninteresting image distribution, but that its predicate labels are not consistently annotated. This is naturally what we have aimed to fix in our paper. Our experiments show that the relabeled SpatialSense+ is a reliable benchmark to evaluate the spatial relation prediction models, indicating our contribution to the whole community.

Q5: The conclusion that language bias is removed cannot be drawn from the experiment result that Bbox-only is better than Bbox+language.

Answer: We are sorry for the incorrect statement. There are two kinds of language bias:

(1) The language bias from commonsense: a computer is usually on a table, so the model will predict “on” without adequately considering the image and bounding box information, even when the computer is actually under the table. Actually, this kind of language bias has been mitigated by the original SpatialSense dataset through adversarial crowdsource strategy. And our SpatialSense+ is relabeled based on the original dataset and thus inherits this property. Consequently, bbox+language performs worse than bbox-only in both SpatialSense+ and SpatialSense datasets. We will correct this point and ensure clarity in our revision.

(2) The language bias from some idiomatic expressions, i.e., the problem we discussed in Section 3.2, which will lead to ambiguous and noisy annotation, e.g., the boy is sometimes labeled as in snow and sometimes on snow. To illustrate this, we show some examples in the tables below, where language biases impact labeling. For example, “book-on-shelf” samples are actually False because the books were not placed on the top of the shelves, but in one of the tiers of the shelf. These samples cannot be predicted only according to language and only affect the annotation quality. During our relabeling process, these mistakes are corrected by our precision and physically grounded definitions. These examples underscore our success in eliminating idiomatic expression language bias. We will include this analysis in our revision.

On:

In:

We appreciate your precious time. We are eager to engage in further discussions to clear out any confusion.

Dear Reviewer,

Thanks again for your precious time and insightful suggestions. We hope we have addressed your concerns comprehensively. As the deadline is approaching, with less than 8 hours remaining, we kindly request that you let us know if there are any further questions or clarifications needed.

Best regards,

Authors

Thank you very much for the insightful suggestions and supportive comments. We hope the following response will address your concerns.

Q1: Compare with large vision language models.

Answer: Thank you for noting that our work precedes the publicly available large vision-language foundation models. It is indeed valuable to evaluate these models on spatial relationship prediction extensively, and we have performed some early tests on LLaVA. We evaluate the SpatialSense+ dataset on LLaVA latest version, i.e., LLaVA-v1.5-13b, and it only gets 50.27% overall accuracy, indicating that LLaVA cannot handle this physically grounded spatial relation task. We think this result is very interesting and provides some inspiration for us about future work, so thank you very much again for this insightful suggestion.

Regardless of the specific results, for the foreseeable future, computation and time costs make it impractical for these foundation models to be directly applicable in many scenarios of interest, such as robotics with size, weight, power, and real-time operation constraints. We believe that it is valuable both practically for these applications and scientifically to understand and maximize the computer vision capabilities of smaller models for a large variety of tasks including ours.

Q2: The annotation of SpatialSense+ can be very hard to scale.

Answer: Thanks for your insightful question. We do not expect the labeling process to scale up, but want to fix this issue by taking advantage of large-scale unsupervised data and few-shot labeled data (i.e., self-supervised pretraining and finetuning). We admit that annotating such complex tasks cannot easily scale up, because it would definitely be very difficult and cost a lot of labor and financial resources. However, with the recent progress of foundation models, finetuning from large pretrained models makes it possible to inherit rich prior knowledge to handle complex downstream tasks with few-shot training sets. This has been a common solution for difficult problems in deep learning nowadays. Motivated by this, in this paper, we aim to fine-tune a good relation prediction model from the pretrained models with a few samples. In the future, especially in the age of large foundation models, we expect the solution to be reading out the relation information from foundation models by few-shot finetuning or even zero-shot prompting. Therefore, we do not require a large-scale spatial relation dataset, but only need a good pretrained model and a well-designed finetuning architecture and strategy, which is actually what we investigate in this paper.

Once again, we are grateful for the valuable and supportive comments!