Hierarchical Multi-Grained Reasoning for Object Concept Learning

We thank reviewer c9Bi for the valuable and constructive comments. We address the concerns as follows.

Q1：How the imbalance ratio influences the model performance. Does the proposed method work under extremely imbalanced cases?

We agree with the reviewer that there is an imbalance in the distribution of attributes and affordances for objects. The original distribution of attributes and affordances in the OCL dataset is also imbalanced. To illustrate the impact of the imbalance ratio on our method, we set the same imbalance ratio $r$ for both attributes and affordance, and we conduct experiments as follows:

Table 1: Ablation of imbalance ratio $r$ on OCL. The values in "()" represent the imbalance ratio.

By setting the same imbalance ratio in the OCL baseline OCRN and our method HGR, it can be observed that our method still improves performance. This indicates that our approach can effectively work under extremely imbalanced cases.

Q2：The authors mainly compare the results with one work, which makes the results not very convincing.

To demonstrate the effectiveness of our method, we replace CLIP with the LLaVA-1.6 framework for validation, and the results are shown as:

Table 2: OCL accuracies with LLaVA-1.6 as baseline.

The results indicate that our method can enhance the existing vision-language models' ability to recognize object attributes and affordances. Furthermore, in Section 4.2, our method is applied to multi-task indoor scene understanding and weakly supervised affordance grounding tasks, and the accuracy is also improved significantly. This phenomenon further demonstrates the effectiveness of our approach.

Q3：What should we do if these annotations are unavailable?

In the OCL setting, the ground-truth bounding boxes are employed. We replace the bounding boxes with the predicted boxes extracted by Faster R-CNN. The results of our experiments are shown in Table 3. From the experimental results, it can be observed that there is a decline in performance after using the pre-trained Faster R-CNN to extract the prediction boxes. However, our method can still enhance the performance of the baseline.

Table 3: Ablation of ground-truth bounding boxes (bbox) on OCL.

Thanks for the response. I have reviewed the comments from other reviewers, and the author's explanation partially addresses my concerns. I would maintain my rating.

We thank Reviewer 3E1s for the valuable comments.

Q1: It does not demonstrate how the proposed method addresses the many-to-many problem.

Although multi-label recognition is not a new problem, it is under explored in the Object Concept Learning (OCL) task. Multi-label recognition mainly focus on the mapping between images and categories. The main challenge of OCL lies in the many-to-many mapping relationships between objects and concepts. That is, an object could have multiple different attributes and affordances (concepts). Meanwhile, an attribute and affordance could also belong to multiple different objects. To overcome this, we propose a Hierarchical Multi-Grained Reasoning framework. For objects in an image, an attribute and affordance could belong to multiple different objects, which means concepts exist across objects. Thus, we first design object-agnostic prompts to enhance concepts-objects mapping precision. Subsequently, a mapping between objects-concepts is formed by integrating contextual global information and fine-grained local information. Furthermore, to make the mapping between objects and concepts more precise, we introduce counterfactual reasoning to identify causal relationships between certain attributes and affordances. This enhances the mapping connections between objects and these concepts, which in turn improves recognition performance. Significant performance improvements and extensive visual analysis have demonstrated the superiority of HGR.

Q2: The generalizability of our method.

To ensure a fair comparison with existing methods, we follow the same settings of the OCL [ref1] task, utilizing ground-truth bounding box and causality annotations. The experimental results in Section 4 demonstrate the effectiveness of our approach. Furthermore, to validate the generalizability of our proposed method, we present our method application in Multi-task Indoor Scene Understanding and Weakly Supervised Affordance Grounding tasks in Section 4.2. For fairness, we maintain the same settings as the baselines in these tasks, without grounding-truth bounding box and causality annotations. In Weakly Supervised Affordance Grounding task, our method outperforms the SOTA method by 5.2%. In the challenged Multi-task Indoor Scene Understanding task, our method outperforms the SOTA method by 0.9%. In addition, we also evaluate our method in the medical dataset, which can be seen in Reviewer c9Bi Q4. The experimental results show that our method could enhance the prediction accuracy, highlighting the generalizability of our approach.

[ref1] Beyond Object Recognition: A New Benchmark towards Object Concept Learning, ICCV 2023

Q3: Explanation of problem definition.

The values of $C_{\alpha}$ and $C_{\beta}$ are initialized using the visual features $F_{a}$ . Through attention-based aggregation, they are progressively updated. Here, $\alpha$ and $\beta$ represent attributes and affordances, respectively, which correspond to the subscripts $h_{\alpha}$ and $h_{\beta}$ in Equation (1).

Q4：Expression issue.

Thank you for pointing out the expression issues. We will make corrections in the revised version.

Q5: Please clarify the difference between object concept learning and multi-label object recognition. Is it feasible to treat concepts directly as categories?

Object Concept Learning (OCL) task aims to push the envelope of object understanding rather than just recognizes object categories. It requires machines to reason out attributes and affordances, and simultaneously give the reason: what attributes make an object possess these affordances [ref1].

Attributes and affordances should not be directly treated as categories. Category apple is a symbol indicating its referent (real apples). Attributes depict object states [ref2]. An elegant characteristic of attributes is cross-category: objects of the same category can have various states (big or fresh apple), while various objects can have the same state (sliced orange or apple). Affordance indicates what actions humans can perform with objects, and it is also cross-category. If the category is the first level of object concept, the attribute can be seen as the second level closer to the physical fact. And affordance can be seen as the third level, which is closely related to common sense and causal inference [ref3].

[ref1] Beyond Object Recognition: A New Benchmark towards Object Concept Learning, ICCV 2023
[ref2] Discovering states and transformations in image collections, CVPR 2015
[ref3] The ecological approach to the visual perception of pictures. Leonardo 1978

We thank reviewer AyMp for the valuable and constructive comments. We address the concerns as follows.

Q1：The novelty of the method.

Our main contribution is not another implementation of the OCL baseline, but proposing a Hierarchical Multi-Grained Reasoning framework. This framework leverages vision-language model to progressively guide the mapping of many-to-many relations between objects and concepts (object-concepts and concept-objects). Additionally, it introduces counterfactual reasoning to refine the mapping relations further, enabling the model not only to identify attributes and affordances but also to understand the causal relationships between them. Current methods for improving the accuracy of many-to-many mappings primarily focus on enhancing the discriminative features of objects. In this work, we first explore designing a dedicated reasoning method for learning object concepts, achieving promising results.

Q2: Explanation of causal relations.

We first clarify that we construct the counterfactual reasoning process during training. By imposing interventions on attribute prompts that have attribute-affordance causal relationships and designing counterfactual losses, the model could learn causal relationships. The improvement in causality metrics $S_{\mathbf{ITE}}$ and $S_{\mathbf{α-β-ITE}}$ prove the effectiveness of our method. To further demonstrate the effectiveness of our method, we conducted experiments on the zero-shot causal setting, and the results are as follows:

Table 1: Performance results for OCL dataset on zero-shot causal learning task.

The results indicate that our method can enhance the model's reasoning performance. In this setting, 300 attributes-affordances causality annotations are used as unseen causal relations. 785 attributes-affordances causality annotations are used as seen causal relations. More details will be updated in the revised version.

Q3: The model size seems considerably larger than that of the baselines. A comparison would be appreciated.

Thanks for your suggestions. The model size is indeed larger than other baselines. We compare our method HGR with the baselines, and the results are as follows:

Table 2: Statistics of Param.

Q4: Additional baseline comparision.

Thanks for your suggestions. We have made our best efforts to conduct experiments on LLaVA-1.6 as additional baseline, and the results are as follows:

Table 3: OCL accuracies with LLaVA-1.6 as baseline.

The results indicate that our method can enhance the existing vision-language models' ability to recognize object attributes and affordances.

Q5: Minor typos

Thank you for pointing out the relevant issues. We will make corrections in the revised version.

We thank reviewer U8nc for the valuable and constructive comments. We address the concerns as follows.

Q1: The causal relationships between attributes and affordances.

We agree with the reviewer's viewpoint that not all attributes of the object have causal relationships with affordances. In the OCL setting and the OCL dataset construction, this problem has been considered and solved by choosing about 10% convinced attribute-affordance classes as attribute-affordance causal pair candidates [ref1]. Thus, as described in Section 3.2.2, we design counterfactual reasoning based on the causality annotation. Additionally, the causality pairs shown in Figure 5 exist in the causal annotations of the dataset. To address these concerns and clarify this point further, we will clarify it in the revised version.

[ref1] Beyond Object Recognition: A New Benchmark towards Object Concept Learning, ICCV 2023

Q2: Multi-grained representation and augmented representation for robust representation are common in deep neural networks.

While multi-grained and augmented representations are common in deep neural networks, these approaches have yet to be well explored for OCL tasks. The OCL task not only identifies the higher-level knowledge attributes and affordances of objects but also reasons out the causal relationships between specific attributes and affordances. Our method is not merely a simple multi-grained feature augment approach; instead, we design a hierarchical reasoning framework that integrates contextual information and causality to reason out higher-level knowledge attributes and affordances. Moreover, since causal relationships exist between specific attributes and affordances, we introduce a counterfactual reasoning module to help the model more precisely capture these causalities during training, thereby improving concepts' recognition performance. This is not just a simple feature augmentation but a facilitation for the model to infer the causal relationships behind affordances. Experimental results validate the effectiveness of our method.

Q3：The demonstration of causal relations.

We adopt causal metrics $S_{\mathbf{ITE}}$ and $S_{\mathbf{α-β-ITE}}$ for evaluation in experiment part Table 1 to demonstrate the extent to which the model learned causal relationships. $S_{\mathbf{ITE}}$ evaluates the model's causal reasoning ability by measuring the attribute-affordance causality prediction results. $S_{\mathbf{α-β-ITE}}$ means multiplying $S_{\mathbf{ITE}}$ with predicted probabilities. It can be seen that our method has achieved improvements on both of these metrics.

Additionally, to qualitatively show the causal relation between specific attributes and affordances, we will upload our visualization results in Appendix A.8 of the revised version. Particularly, our visualizations in Figure 3 and Figure 6 just aim to show that the model could progressively focus on attribute and affordance features through hierarchical reasoning.

Q4: Do you think current MLLM model can easily address affordance and attributes recognition?

We think that MLLM has the potential to address the problem of attribute and affordance recognition, but MLLM still faces some challenges. For instance, in [ref1], [ref2], and [ref3], MLLM is used as an external knowledge base to provide some attribute (small, long) and affordance (hold, grasp) knowledge. However, classic MLLM is generally trained on generic image-text pairs, lacking critical robotic knowledge to understand object affordances and their physical properties [ref4] [ref5]. For example, the machine needs to understand objects' geometric structure to predict the movable contact regions. Enabling MLLM to solve the recognition of attributes and affordances easily still requires sustained attention and research.

[ref1] ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction, arxiv 2024
[ref2] INTRA: Interaction Relationship-aware Weakly Supervised Affordance Grounding, ECCV 2024
[ref3] WorldAfford: Affordance Grounding based on Natural Language Instructions, arxiv 2024
[ref4] ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models, arxiv 2024
[ref5] ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation, CVPR 2024

Thanks for your response and the provided references. The annotated causal relations are questionable. I think it relies on a strong assumption which the paper did not claim. By the way, fresh'' does not cause eatable'', even if it is limited to the figures in the paper. For the use of these keywords, concepts, and claims, I'd like to refer to Judea Pearl's comments about the recent use of those concepts. Though I think it is confusing about the causality, I keep boardline reject given that this paper has done extensive experiments,