Attribute Recognition with Image-Conditioned Prefix Language Modeling

We want to first thank the reviewers for recognizing our work’s novel reformulation of the attribute learning problem and its contribution in successfully leveraging pretrained large language models for fine-grained visual understanding through generative prompting. We also thank the reviewers for recognizing the significance of our experimental results, which 1) shows strong performance on par with SOTA, despite not having task-specific modules or pretraining with annotated in-domain datasets, and 2) supports our claim that CLIP representation alone is ill-suited for attribute learning and that our proposed generative prompting can bridge the gap in performance. And lastly, we thank the reviewers for acknowledging our benchmark’s broad usefulness to the community, as it unifies object and attribute recognition into a single setting and allows for direct comparisons that can measure the level of disentanglement between attribute and object representations.

Q1: Thank you for the comment. We want to clarify that the main insight of our work is the formulation of attribute learning as a knowledge retrieval problem to extract relevant knowledge embedded in large vision-language models (VLM) through the means of generative prompting. We also establish the fundamental connection between conditional probability graphs and prompt template construction in the context of prefix language models, which we illustrate in figure 2, and recognize its importance and applications for visual attribute learning. To further elaborate, this formulation allows us to flexibly extract relevant knowledge of object-attribute-image relationships from the pretrained foundational models for downstream tasks outside of the foundation model settings, as we mentioned in paragraph 4 of the introduction. To our knowledge, we are the first to propose this novel view of treating attribute learning as a language modeling and knowledge retrieval problem and in applying VLMs towards visual attribute recognition, as we discussed in paragraph 4 of Related Work.

To further highlight the difference between our work and other captioning work in visual understanding, we would like to reiterate parts of the Approach section in the paper. One of our central ideas is to formally express attribute learning as a task of learning image-object-attribute conditional probabilities, which naturally leads to generative prompting in a learning distillation paradigm - constructing custom probability dependencies between objects and attributes at inference time (with 4 examples given in section 3.3 and in figure 2) and use it to extract relevant knowledge on image-object-attribute relationship acquired during standard large-scale pretraining in the foundation VLM model, as we discussed in paragraph 6 in section 3.3 and illustrated in figure 1.

Q2: Thank you for the comment. We will include additional discussions and citations on compositional zero-shot learning in the final submission and discuss how it relates to attribute learning and our work.

Q3: We have included numerous non-LLM baselines in our experiment table for comparison. We would love to add additional important ones we have not yet included if you could kindly bring them to our attention.

Q4: We will expand and elaborate the method section in our final submission.

Q5: We examined a list of potential dataset and found HICO and V-COCO to be more relevant, but not necessary for expanding on for our work’s central idea due to overlapping domain (COCO) and the focus on relational attributes.

Q6 Q7: We will fix these issues in the final submission.

Q8: LAION is a widely-accepted and widely-used dataset for pretraining, so our method would be considered as standard and fair.

Q9: Thank you for the comment. Our model performs significantly better on the less frequent categories in the distribution long tail, as shown in table 3. SCoNE and TAP’s slightly higher overall mAP and head mAP does not necessarily mean they are superior models, since they represent performance on frequently observed categories where it is easy for models to fit to the dataset. Our model’s performance on the long tail shows that it fits more to the prior knowledge carried by the foundation models.

Please see below for qualitative results on the least frequent attributes in VAW:

We want to first thank the reviewers for recognizing our work’s novel reformulation of the attribute learning problem and its contribution in successfully leveraging pretrained large language models for fine-grained visual understanding through generative prompting.. We also thank the reviewers for recognizing the significance of our experimental results, which 1) shows strong performance on par with SOTA, despite not having task-specific modules or pretraining with annotated in-domain datasets, and 2) supports our claim that CLIP representation alone is ill-suited for attribute learning and that our proposed generative prompting can bridge the gap in performance. And lastly, we thank the reviewers for acknowledging our benchmark’s broad usefulness to the community, as it unifies object and attribute recognition into a single setting and allows for direct comparisons that can measure the level of disentanglement between attribute and object representations.

Q1: Thank you for your comment. We want to clarify that the main contribution of our work is not the prompt engineering, but the formulation of attribute learning as a knowledge retrieval problem to extract relevant knowledge embedded in large vision-language models (VLM) through the means of generative prompting. Our work advances the state of the art by reframing the attribute learning problem to solve it through leveraging standard pretrained VLMs. We also establish the fundamental connection between conditional probability graphs and prompt template construction in the context of prefix language models, which we illustrate in figure 2, and recognize its importance and applications for visual attribute learning. To further elaborate, this formulation allows us to flexibly extract relevant knowledge of object-attribute-image relationships from the pretrained foundational models for downstream tasks outside of the foundation model settings, as we mentioned in paragraph 4 of the introduction. To our knowledge, we are the first to propose this novel view of treating attribute learning as a language modeling and knowledge retrieval problem and in applying VLMs towards visual attribute recognition, as we discussed in paragraph 4 of Related Work.

Q2: Thank you for your comment. To reiterate section 3.1 on page 4, generative prompting is better than contrastive prompting because it explicitly decomposes and represents the dependency in text input by taking word order and relation into consideration, which also align closer with a prefixLM’s pretraining-acquired knowledge of diverse compositions of object-attribute dependencies. Given a nonsensical pair like “sky is parking”, the conditional probability term for “parking”, P(“parking” | v, s0, “sky”, “is”), would make the joint probability very small. CLIP-based contrastive prompting does not model dependencies in the text and has the problem of discrepancy between pretraining objective and downstream tasks, as it learned to align text to image without structures, resulting in a focus on objects, but is later tasked with understanding finer attributes that were not salient in pretraining objectives.

Q3: Thank you for the comment. Our model performs significantly better on the less frequent categories in the distribution long tail, as shown in table 3. SCoNE and TAP’s slightly higher overall mAP and head mAP does not necessarily mean they are superior models, since they represent performance on frequently observed categories where it is easy for models to fit to the dataset. Our model’s performance on the long tail shows that it fits more to the prior knowledge carried by the foundation models.

Additionally, the methods should not be directly compared because our focuses on cross-domain knowledge extraction and learning distillation, instead of constructing a task-specific model with specialized modules and training procedures. SCoNE and TAP both rely on object mask supervision and custom datasets during training, require task-specific modules, and are trained and tested in the same domain.

Q4: Thank you for the comment. CoCa was selected because it integrates both CLIP based training as well as captioning based training in the same model, allowing for a fair comparison, and because we can finetune it for comparisons with baselines.

Table 4 and 5 highlights the difference between probability metamodels in a zero-shot setting. Most of the baselines do not have zero-shot capability. TAP does (embedding space projection, similar to contrastive prompting), but its implementation is not available to us.

Thank you for the list of potential datasets we can explore. We carefully examined each dataset and found HICO and V-COCO to be more relevant, but not necessary for expanding on for our work’s central idea due to overlapping domain (COCO) and the focus on relational attributes.

We want to thank the reviewer for recognizing our work’s novel reformulation of the attribute learning problem and its contribution in successfully leveraging pretrained large language models for fine-grained visual understanding through generative prompting. We agree that the paper’s key idea of using generative prompting/modeling to solve complex localized reasoning tasks is an interesting and promising direction of exploration for the community to explore. We also thank the reviewers for recognizing the significance of our experimental results, which 1) shows strong performance on par with SOTA, despite not having task-specific modules or pretraining with annotated in-domain datasets, and 2) supports our claim that CLIP representation alone is ill-suited for attribute learning and that our proposed generative prompting can bridge the gap in performance. And lastly, we thank the reviewers for acknowledging our benchmark’s broad usefulness to the community, as it unifies object and attribute recognition into a single setting and allows for direct comparisons that can measure the level of disentanglement between attribute and object representations.

We want to first thank the reviewers for recognizing our work’s novel reformulation of the attribute learning problem and its contribution in successfully leveraging pretrained large language models for fine-grained visual understanding through generative prompting. We agree that the paper’s key idea of using generative prompting/modeling to solve complex localized reasoning tasks is a promising direction of exploration. We also thank the reviewers for recognizing the significance of our experimental results, which 1) shows strong performance on par with SOTA, despite not having task-specific modules or pretraining with annotated in-domain datasets, and 2) supports our claim that CLIP representation alone is ill-suited for attribute learning and that our proposed generative prompting can bridge the gap in performance. And lastly, we thank the reviewers for acknowledging our benchmark’s broad usefulness to the community, as it unifies object and attribute recognition into a single setting and allows for direct comparisons that can measure the level of disentanglement between attribute and object representations.

Q1: Thank you for your comment. We want to clarify that the main contribution of our work is not the prompt engineering, but the formulation of attribute learning as a knowledge retrieval problem to extract relevant knowledge embedded in large vision-language models (VLM) through the means of generative prompting. Our work advances the state of the art by reframing the attribute learning problem to solve it through leveraging standard pretrained VLMs. We also establish the fundamental connection between conditional probability graphs and prompt template construction in the context of prefix language models, which we illustrate in figure 2, and recognize its importance and applications for visual attribute learning. To further elaborate, this formulation allows us to flexibly extract relevant knowledge of object-attribute-image relationships from the pretrained foundational models for downstream tasks outside of the foundation model settings, as we mentioned in paragraph 4 of the introduction. To our knowledge, we are the first to propose this novel view of treating attribute learning as a language modeling and knowledge retrieval problem and in applying VLMs towards visual attribute recognition, as we discussed in paragraph 4 of Related Work.

To further address your concern on whether our work develops a new method or advances the state of the art, we would like to reiterate parts of the Approach section in the paper. We formally express attribute learning as a task of learning image-object-attribute conditional probabilities, which can be solved in a novel pretraining-to-distillation paradigm with our proposed method. Our method constructs custom probability dependencies between objects and attributes at inference time (with 4 examples given in section 3.3 and in figure 2) and use it to extract relevant knowledge on image-object-attribute relationship acquired during standard large-scale pretraining in the foundation VLM model, as we discussed in paragraph 6 in section 3.3 and illustrated in figure 1. This theoretical and practical reframing of the attribute learning problem and the proposed method of applying and extracting knowledge from LLMs towards the task through generative prompting are both novel and contribute to advancing the state of the art.

Q2: Thank you for your comment. We invested a large amount of time but did not find any public, related work on prompt engineering in the context of extracting knowledge from vision-language models to improve attribute learning problems. To clarify, our work is not a paper on prompt engineering, since its central idea is to introduce a new way of thinking about the attribute learning problem through a probabilistic modeling perspective, as we first discussed in paragraph 4 of Related Work and expanded upon in paragraph 6 of section 3.3 in Approach. As part of the demonstration of our formulation and method’s capabilities, we study several possible conditional probability graphs at inference time, expressed through different constructions of the generative prompt. We conduct thorough experiments in section 4.2 and 4.3 to show how model performance can be improved by just improving how knowledge is extracted from the foundation model, namely by studying and improving the conditional metamodel we use to retrieve information. Lastly, we also want to mention that besides zero-shot experiments, our work also dedicated half of section 4 on fine-tuning experiments, which goes beyond what prompt engineering papers study.

Q3: Thank you for the comment. As we also discussed above, the prompt study itself is not the focus. Our main contribution remains to be the reframing of the attribute learning problem and the proposed method of extracting and applying LLM knowledge towards the task.