Predicate Hierarchies Improve Few-Shot State Classification

Q: Object-centric encoder for environments with significant distribution shifts.

A: We see empirically that PHIER’s object-centric encoder performs well even on environments with significant distribution shifts, such as CALVIN. In Figure 8 in the Appendix, we add an example of how the encoder localizes objects in CALVIN. To adapt to environments with even larger distribution shifts where the performance may decrease, we note that PHIER’s object-centric encoder can be finetuned with more data as well. We have included this example and discussion in Appendix Section A.8.

Q: Additional visualizations of the learned Poincare Ball.

A: Thank you for your suggestion. We added visualizations of the joint image-predicate space for BEHAVIOR on the Poincaré disk in Figure 6 in the Appendix, highlighting the hierarchical semantic structure captured by PHIER's embeddings. By grouping the joint image-predicate embeddings by predicate, we uncover the inferred predicate hierarchy. For instance, we see that embeddings for NextTo are positioned closer to the origin compared to those for OnLeft, accurately reflecting their hierarchical relationship — OnLeft is a more specific case of NextTo. Furthermore, embeddings for Touching are nearest to the origin, consistent with its role as the most general predicate. For example, when one object is Inside or OnTop of another, they are inherently Touching. Similarly, objects that are NextTo or OnLeft are also frequently Touching. This visualization demonstrates that PHIER captures not only semantic structure but also nuanced hierarchical relationships between predicates.

We further analyze the embeddings for novel predicates after few-shot learning with only five examples. Notably, even with such limited data, PHIER successfully integrates these novel predicates into the latent space and aligns them with their learned counterparts in semantically consistent regions (e.g., OnRight is near OnLeft). By aligning these predicates in similar regions, PHIER is able to leverage its existing knowledge of relevant features for learned predicates (e.g., OnLeft) to reason about novel predicates (e.g., OnRight). This alignment highlights that PHIER effectively encodes the relationships between pairwise predicates in the latent space, enabling generalization to novel predicates with minimal examples. Thanks to your feedback, we have updated Appendix Section A.6 with these visualizations and analyses.

Q: Clarification on ablation details.

A: Thank you for pointing this out! To address your concern, we provide a clear breakdown of our ablation model architectures and explain how we add each component.

Supervised model: We start with a supervised baseline model, which uses an image encoder and text incoder initialized with CLIP and BERT weights, respectively.The embeddings from both encoders are concatenated and passed through a small MLP with three linear layers for classification, and the full model is trained with a binary cross-entropy loss based on the ground truth labels (True or False). We then progressively add each component of PHIER.

+ Object-centric encoder: First, we incorporate the object-centric encoder by replacing the image encoder, text encoder, and concatenation step with our proposed object-centric encoder, while retaining the MLP and loss.

+ Predicate triplet loss: Next, we introduce the predicate triplet loss by adding this term to the total loss function without changing the architecture.

+ Norm regularization loss: We further add the norm regularization loss to get the total loss function with all components, as described in Section 3.4.

+ Hyperbolic metric: Finally, we lift the scene representation to hyperbolic space using an exponential map and replace the first two linear layers in the MLP with two hyperbolic linear layers of the same size. We also use the Poincaré distance metric instead of the Euclidean metric in the self-supervised losses, yielding our final model (PHIER).

Thanks to your feedback, we have added these details about the ablation architectures in Appendix Section C.3. We also added new ablations that remove each component separately in Appendix Section A.4.

Q: Hyperbolic space’s disc area.

A: We provide further details on why the exponential growth of the disc area in hyperbolic space provides a natural and efficient way to represent trees. Note that for a regular tree with a constant branching factor $b$, the number of nodes increases exponentially with the distance from the root, as $(b+1)b^{\mathcal{l}-1}$. We can embed trees in hyperbolic space, as they mirror this exponential growth. For instance, in a two-dimensional hyperbolic space with constant curvature $K = -1$, the circumference of a disc with radius $r$ is $2\pi\sinh r$ while the area of a disc is $2\pi(\cosh r - 1)$. Since $\sinh r = \frac{1}{2} (e^r - e^{-r})$ and $cosh r = \frac{1}{2} (e^r + e^{-r})$, both the circumference and area of the disc grow exponentially with the radius.

This exponential growth allows us to efficiently embed tree structures in hyperbolic space: nodes that are $\mathcal{l}$ levels from the root can be placed on the hyperbolic disc with a radius proportional to its level $\mathcal{l}$, while nodes less than $\mathcal{l}$ levels within the sphere. Thus, we see how this property allows hyperbolic space to serve as a continuous representation of discrete trees. Thanks to your feedback, we have added these details to Appendix Section C for further clarification.

[1] Gromov, Mikhael. "Hyperbolic Groups." Essays in Group Theory. New York, NY: Springer New York, 1987. 75-263.

[2] Dyubina, Anna, and Iosif Polterovich. "Explicit Constructions of Universal ℝ-Trees and Asymptotic Geometry of Hyperbolic Spaces." Bulletin of the London Mathematical Society 33.6 (2001): 727-734.

[3] Nickel, Maximillian, and Douwe Kiela. "Poincaré embeddings for learning hierarchical representations." Advances in Neural Information Processing Systems 30 (2017).

[4] Chami, Ines, Albert Gu, Vaggos Chatziafratis, and Christopher Ré. "From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering." Advances in Neural Information Processing Systems 33 (2020): 15065-15076.

Q: Evidence that large vision models struggle with spatial relationships.

A: We highlight several recent works which have explored the limitations of large vision models (VLMs) in spatial understanding. Notable examples include Liu et al. [1], which systematically analyzes the weaknesses of VLMs in spatial reasoning tasks, and Tong et al. [2], which demonstrates that multimodal LLMs often fail to encode spatial relationships effectively. Additionally, this remains an active area of research, as explored in works like Chen et al. [3], Fu et al. [4], and Cheng et al. [5]. In our paper, results on CALVIN and BEHAVIOR benchmarks further support these findings, showing significant room for improvement in understanding these predicate relationships.

We also note our task takes a single image as input, while methods that conduct multi-perspective image processing generally require multiple viewpoints. Methods that generate a full explicit 3D scene from a 2D image such as Sargent et al. [6] takes hours to lift images into 3D, and still require subsequent object detection and 3D classifier training for predicate classification. Moreover, these methods struggle on scenes that contain multiple objects, which many of the scenes in our test sets contain.

[1] Liu, Fangyu, Guy Emerson, and Nigel Collier. "Visual Spatial Reasoning." Transactions of the Association for Computational Linguistics 11 (2023): 635-651.

[2] Tong, Shengbang, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578. 2024.

[3] Chen, Boyuan, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. "SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465. 2024.

[4] Fu, Xingyu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. "Blink: Multimodal Large Language Models Can See But Not Perceive." In European Conference on Computer Vision, pp. 148-166. Springer, Cham, 2025.

[5] Cheng, An-Chieh, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. "SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model." arXiv preprint arXiv:2406.01584 (2024).

[6] Sargent, Kyle, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan et al. "ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9420-9429. 2024.

Q: Evaluation on ID-OOD and real world samples.

A: Thank you for the suggestion. We note that pre-trained models are typically trained on large-scale real-world datasets with vast amounts of diverse data (e.g., from our comparisons, BLIP-2 was trained on 129M images, ViperGPT composes several large, pre-trained models such as GLIP [1] and X-VLM [2], and GPT-4v was trained on an internal unspecified large-scale dataset). Hence, these models inherently do not differentiate between ID and OOD scenarios, as their training data overlaps significantly with both. We agree that reporting results on ID-OOD is more comprehensive, hence thanks to your suggestion, we added results for pre-trained models on ID-OOD. We see that pre-trained models demonstrate similar performance ID and OOD, and PHIER still significantly outperforms all models (e.g., by 33.6% on OOD CALVIN and 11.4% on OOD BEHAVIOR). These results reveal no consistent trend between ID and OOD scenarios for pre-trained models. Notably, the performance drop for PHIER between ID and OOD is comparable to that of the pre-trained models, demonstrating that our model's generalization behavior is within a reasonable range.

Similarly, we add in additional results of the pre-trained models on real-world samples, while noting that we consider these models our upper bound (due to their immense real-world train data), compared to PHIER and other supervised works. We compare the pre-trained models against PHIER tested under zero-shot and few-shot settings. In the zero-shot setting, we see that PHIER outperforms ViperGPT and BLIP-2 by 6.0% and 1.4% respectively, showing the potential for a small model trained on significantly less data, to reach the performance level of large pre-trained models. However, GPT-4v outperforms PHIER by 10.4%. which we hypothesize is due to its model size and dataset scale, even compared to prior pre-trained works. In the few-shot setting using only two examples, PHIER’s performance improves significantly and narrows the gap with GPT-4v to just 0.9%. This further demonstrates that PHIER’s inferred predicate hierarchy enables it to generalize to novel queries with efficient adaptation.

Thanks to your feedback, we have included both of these experiment results in Section 5 in the main text.

[1] Li, Liunian Harold, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang et al. "Grounded Language-Image Pre-Training." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975. 2022.

[2] Zeng, Yan, Xinsong Zhang, and Hang Li. "Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts." arXiv preprint arXiv:2111.08276 (2021).