Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem

Thank you for your detailed review and valuable feedback. We appreciate the opportunity to address your comments and clarify our work. Below is our point-by-point response:

Concerns about response parsing

We thank the reviewer for pointing out this important concern. We have now eliminated the intermediate parsing step and ask the models to directly provide its response as output. We find that all of the same binding errors are present without secondary response parsing (see global rebuttal).

Explanation of metrics used in Scene Description task

We have updated our scene description task to explicitly evaluate the relationship between feature interference across objects and model performance. Instead of using entropy, we now estimate the "likelihood of a swap error," defined as the number of triplet pairs in the color/shape feature matrix where swapping two of the triplet features results in a binding error. We find that this new metric is significantly correlated with task performance, and provides a more elegant account of model behavior. See section A and B of the attached PDF for the updated results. Our dataset is generated by balancing the number of trials across each object count and feature diversity condition. Detailed descriptions of how this novel metric is computed and how our scene description dataset was generated are provided in the updated manuscript.

Edit distance is defined as the number of objects the model failed to identify from the ground truth set of objects plus the number of objects erroneously identified (conjunction of shape/color for identified object not present in the ground truth). We have added a detailed description of the updated prompting strategy and a definition of edit distance in the revised manuscript.

Improved plotting for scene description task:

We appreciate the suggestion. We have implemented these changes in the updated plots to improve clarity (see section A of the attached PDF).

Explanation of RMTS task:

We have clarified that the relations are ‘same shape, different color’. The correct answer is Target Pair 2 because it satisfies both relations, while Target Pair 1 only satisfies one. We did not observe any evidence that the task was ambiguous for the VLMs. This clarification has been added to the revised manuscript.

Feasibility of running decomposed trial conditions on tasks other than RMTS

For counting and scene description, we believe that the decomposed condition would likely improve performance, but we did not test this because we felt that it may be viewed as trivial. For example, in the counting task, this would just amount to counting the number of images that are presented. For the search task, the number of objects exceeds the context window size for images (10 images max for GPT-4V).

Limitations of Closed Source Models

We acknowledge this limitation and have added experiments with LLaVA 1.5. While LLaVA did not respond consistently with our task instructions using the original prompts, modified prompts show that the subitizing limit in the counting task is preserved (see section C in attached PDF). However, the model struggles with search tasks (section F) and visual analogy. The absence of effects in LLaVA may be due to performance differences or prompting strategy.

We hope these revisions and clarifications address your concerns. Thank you again for your insightful feedback.

I thank the authors for their rebuttal. My concerns were quite minor in the first place, and mostly related to presentation. Beyond responding to my own concerns, given that the authors have substantially expanded the paper with more thorough analyses, I have increased my score by one point.

Thank you for your detailed review and valuable feedback. We appreciate the opportunity to address your comments and clarify our work. Below is our point-by-point response:

Concerns about generalizability and scope of our results

We thank the reviewer for suggesting to run additional control experiments and to analyze a broader range of models. We have run the following additional experiments to strengthen our work.

1. We have expanded our analysis to include five VLMs: GPT-4v, GPT-4o, Claude Sonnet 3.5, Google Gemini Ultra 1.5, and LLaVA 1.5. Additionally, we have expanded the text-to-image model experiments to include four models: DALL-E-3, Stable Diffusion Ultra, Google Parti, and Google Muse. The general effects are present across all models, with some minor deviations for LLaVA, which we suspect is due to its specific limitations relative to stronger open source models.

2. For the visual search task, we included a condition where the color of the target and distractors varies across each trial. The results show that model performance is comparable to the original experiment, with the same qualitative effect. Please refer to sections E and F of the attached PDF for the figure of results in an example model.

3. For the counting task, we introduced conditions where we manipulate the color and shape dimensions of the stimuli. Our findings support our hypothesis that capacity limits arise due to issues with representational interference. Performance is generally highest in the highest entropy condition (unique colors and shapes across all objects), lowest in the low entropy condition (uniform colors and shapes), and intermediate for the medium entropy conditions (either unique colors or shapes, but not both). Please refer to section C of the attached PDF for the figures of the updated results.

Evidence for compositional representations in VLMs/LLMs

The reviewer has requested that we cite evidence in support of the claim that VLMs learn compositional representations. We have included references to the following papers, which provide evidence that LLMs/VLMs form compositional representations (Lepori et al. 2023; Lewis et al. 2022; McCoy 2022; Yun et al. 2022, Yu et al. 2023), and agree that future work should directly investigate the representations themselves. We additionally note that if VLMs did not learn compositional representations, we would not expect to observe a different pattern of errors in the model behavior. The binding problem arises only when a system uses compositional representations, thus evidence for the binding problem can be viewed as indirect evidence for the presence of compositional representations.

Solving the binding problem with improved prompting?

It is difficult to say for sure, given the lack of transparency about closed-source model architectures, whether there might be some prompting approach that improves results. However, there are reasons to believe that prompting alone cannot resolve the binding problem in current models. First, we have already attempted to prompt the models to enumerate objects one at a time, which did not improve performance (indeed, the performance reported in both counting and search tasks is prompting the model to explicitly enumerate all objects). Second, there are architectural limitations:

1. VLM image encoders may struggle to maintain distinct object representations within a single embedding. Compressing multi-object scenes while preserving accurate feature bindings is challenging, potentially limiting the model's ability to process objects individually.

2. In autoregressive transformers, it is possible to perform some degree of serial processing through the output of the model (i.e. chain-of-thought), but in order to use this to sequentially attend to individual objects, there must be some clear way to index these objects in the model's output, which is often difficult for multi-object scenes.

We have added an explanation of these factors to the revised manuscript and included relevant citations.

We hope these revisions and clarifications address your concerns. Thank you again for your insightful feedback.

Citations:

Lepori, M., Serre, T., & Pavlick, E. (2023). Break it down: Evidence for structural compositionality in neural networks. Advances in Neural Information Processing Systems, 36, 42623-42660.

Lewis, M., Nayak, N. V., Yu, P., Yu, Q., Merullo, J., Bach, S. H., & Pavlick, E. (2022). Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:2212.10537.

McCoy, R. T. (2022). Implicit Compositional Structure in the Vector Representations of Artificial Neural Networks (Doctoral dissertation, Johns Hopkins University).

Yun, T., Bhalla, U., Pavlick, E., & Sun, C. (2022). Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?. arXiv preprint arXiv:2203.17271.

Yu, D., Kaur, S., Gupta, A., Brown-Cohen, J., Goyal, A., & Arora, S. (2023). Skill-Mix: A flexible and expandable family of evaluations for AI models. arXiv preprint arXiv:2310.17567.

Thank you for your detailed review and valuable feedback. We appreciate the opportunity to address your comments and clarify our work. Below is our point-by-point response:

Clarification on human-like capacity limits and the logic of our experiments

The reviewer has asked for clarification regarding human performance in the visual search task, and raised the concern that humans are indeed able to count beyond the subitizing limit when given enough time. Before directly addressing these issues, let us first briefly explain the types of capacity limits that we are interested in, and the conditions under which they arise in human cognition. It is widely acknowledged in cognitive science that there is a distinction between parallel (automatic) and serial (controlled, deliberative) modes of processing (Posner et al. 1980; Schneider et al. 1977; Treisman & Gelade 1980). The parallel mode of processing is dominant when people make rapid judgments, and displays classic capacity limits, such as difficulty counting beyond the subitizing limit, and difficulty in conjunctive search tasks. The serial mode of processing requires temporally extended processing, and is able to overcome these capacity limits, enabling precise counting for large numbers of objects, and serial search to identify specific feature conjunctions.

Our hypothesis in this work is that the current generation of VLMs is primarily limited to the parallel mode of processing. We explain below some reasons for thinking that this might be the case (based on current understanding of the architectures and algorithms used), but, given the close-sourced nature of most current VLMs, our approach in this work is to test whether VLMs display behavioral signatures that resemble rapid parallel processing in human cognition. We find that VLMs do indeed display several specific phenomena that are diagnostic of parallel processing, including 1) difficulty counting beyond the subitizing limit, 2) greater difficulty in counting arrays of objects with uniform features than arrays with variable features (see new results in global rebuttal), 3) performance on disjunctive search tasks that is invariant to the number of objects, combined with decreased performance on conjunctive search tasks as a function of the number of objects, and 4) binding errors in scene description. Taken together, these phenomena provide indirect, but nevertheless highly specific, evidence that VLMs are limited to parallel processing. The fact that humans can complement this form of processing with serial processing – as in counting and serial search – is not at odds with our conclusions. Rather, it suggests a way that we can imagine augmenting VLMs to bring them closer to human performance.

Potential reasons for limited serial processing in current VLMs

We do not claim that serial processing is beyond the ability of any type of vision-language model, including future improvements of these systems, but there are at least two major reasons to suspect that the current generation of VLMs may suffer from difficulty with serial processing:

1. VLM image encoders may struggle to maintain distinct object representations within a single embedding. Compressing multi-object scenes while preserving accurate feature bindings is challenging, potentially limiting the model's ability to process objects individually.
2. In autoregressive transformers, it is possible to perform some degree of serial processing through the output of the model (i.e. chain-of-thought), but in order to use this to sequentially attend to individual objects, there must be some clear way to index these objects in the model's output, which is often difficult for multi-object scenes.

These are certainly not fundamental limitations. It is easy to imagine that serial processing can be improved in VLMs through the use of object-centric visual representations, or the use of recurrent rather than feedforward transformers, and there is indeed some recent work in this direction (You et al. 2023; Webb et al. 2024). Our results suggest that VLMs will benefit from the incorporation of these stronger inductive biases for serial processing. We have added further discussion of these issues to the paper.

Possibility of errors in LLM answer parsing

This is an excellent point. To avoid concerns over response parsing, we have updated our prompts so that VLMs directly return their responses in a form that we can evaluate without a secondary parsing model. We find that all of the same binding errors are present without secondary response parsing (see global rebuttal).

Spike in counting performance for 9 objects

Thanks for highlighting this. We've run extended analysis across more models (see global rebuttal) and don't universally observe this trend. We agree it's interesting – like Rane et al. 2024, we hypothesize it may result from 3x3 grid arrangements in training data.

Citations:

Posner, M. I., Snyder, C. R., & Davidson, B. J. (1980). Attention and the detection of signals. Journal of experimental psychology: General, 109(2), 160.

Rane, S., Ku, A., Baldridge, J., Tenney, I., Griffiths, T., & Kim, B. (2024). Can Generative Multimodal Models Count to Ten?. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 46).

Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information processing: I. Detection, search, and attention. Psychological review, 84(1), 1.

Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive psychology, 12(1), 97-136.

Webb, T., Mondal, S. S., & Cohen, J. D. (2024). Systematic visual reasoning through object-centric relational abstraction. Advances in Neural Information Processing Systems, 36.

You, Haoxuan, et al. "Ferret: Refer and ground anything anywhere at any granularity." arXiv preprint arXiv:2310.07704(2023).

Thank you for your detailed review and valuable feedback. We appreciate the opportunity to address your comments and clarify our work. Below is our point-by-point response:

GPT-4v’s ability to describe synthetic images, and relationship to concurrent work (‘Vision language models are blind’)

Thank you for highlighting this highly relevant concurrent work, which we have now cited in the revised manuscript. We completely agree that VLMs struggle to accurately describe multi-object scenes. In this work, we propose an explanation for why VLMs show this difficulty. Specifically, we propose that it results primarily from an inability to accurately bind features at the object level. This is distinct from explanations that involve hallucinating features that aren’t present in the image (see additional results in section below for an analysis of these failure modes).

Analysis of binding failures vs. hallucinations

We agree that it is important to distinguish binding failures from the hallucination of individual features that aren’t actually present in the image. We have now included an analysis to account for this. We found that the model rarely (<25% of errors) generates descriptions containing features not present in the trial. When this does happen, it is typically due to the misidentification of a feature with a close neighbor (e.g., attributing the color cyan to a blue shape). Most errors in this task can be attributed to binding errors where the model incorrectly attributes a feature to the wrong object.

Clarifications regarding visual search task

In these tasks, subjects are typically asked to identify whether the target is present, so the task is comparable. It is called ‘search’ in the literature, but we agree that it doesn’t necessarily involve localization. We have clarified this in the revised manuscript. We have also clarified why half of the images do not involve targets (because it is really a detection task, there must be some trials without targets in order to obtain a meaningful accuracy estimate).

Regarding the similarity between human and VLM capacity constraints, it is difficult to perform a direct comparison because the original tasks involve a response time measure, and there is no clear analog of response time in VLMs. The classic finding is that response time increases as a function of the number of objects in the conjunctive search task, but is unaffected by the number of objects in the disjunctive search task. Because we do not have a response time measure for VLMs, we instead evaluate accuracy. We find a similar qualitative pattern (accuracy varies as a function of the number of objects in the conjunctive search task, but is unaffected in the disjunctive search task).

We have also added an explanation of how accuracy is calculated in this task. Accuracy is computed as the proportion of trials that are correct. The following accuracy equation has been included in the supplemental for clarity: $acc = \frac{nCorrect_{target} + nCorrect_{distractors}}{nTrials}$

Clarification regarding ‘true description’ and ‘edit distance’ in the scene description task

To address these concerns, in the most recent version of the paper, we have updated the prompts so that the model directly returns a json of the trial objects’ shapes and colors (rather than an open-ended description of the scene). Edit distance here is defined as the number of objects that the model failed to identify from the ground truth json plus number of objects that the model erroneously identified (conjunction of shape/color not present in the ground truth json). We have added a description of the updated prompting strategy and a definition of edit distance in the revised manuscript.

Clarification regarding correct answer in example visual analogy problem

We have clarified that the correct target pair must share both relations (shape and color) with the source pair. The correct answer in the example is Target Pair 2 because it satisfies both the ‘same shape’ and ‘different color’ relations. Target Pair 1 only satisfies one relation. We did not observe any evidence that the task was ambiguous for the VLMs. This clarification has been added to the revised manuscript.

We have also corrected typos identified, and implemented other clarifications (in the caption for figure 2, and the description of the visual search task).

We hope these revisions and clarifications address your concerns. Thank you again for your insightful feedback.