Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects

Thank you for your thoughtful review! Find point-by-point replies below:

Weaknesses:

1. “Related work”: We completely agree! We will use the extra space in the final submission to expand our literature review section to include more insights from both the (vast) cognitive science literature and the mechanistic interpretability literature (see global rebuttal #1 for specific topics). We are also happy to include more related work on the processing pipeline in transformers (including the reference you mentioned).

2. “Theoretical explanation”: We agree, and we are curious to find this out ourselves! While we believe a proper treatment of the learning dynamics that give rise to multi-stage processing in transformers is out of scope for this work, we are highly interested in pursuing this direction in the future. Our current understanding is quoted below and is supported by additional analyses on DINOv2, a vision-only ViT whose pretraining dataset is closer in scale to CLIP than the other models. Despite using a very different training objective, DINOv2 matches CLIP’s performance on discrimination and Relational-Match-to-Sample (RMTS) tasks (99.5% test accuracy on discrimination and 98.2% on RMTS); crucially, it also exhibits similar two-stage processing. We will include these results upon acceptance, though see Figure 1 in the supplemental PDF for an attention pattern analysis on DINOv2.

“Raghu et al. (2021) finds that models pretrained on more data tend to learn local attention patterns in early layers, followed by global patterns in later layers. This might give CLIP a particular incentive to create local object representations, which are then used in relational operations. Future work might test this hypothesis.”

Questions:

1. “Multi-modality”: We were very interested in this question as well. As mentioned earlier, we analyze DINOv2 to explicitly test this. While CLIP does show some differences from DINOv2 (see general rebuttal #2 and the response to reviewer NSpf, question #2), DINOv2 largely recapitulates the results from CLIP. This suggests that data scale drives the adoption of two-stage processing rather than multi-modality.

Citations:

1. Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021). Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34, 12116-12128.

Thank you for your detailed review! Find point-by-point replies below:

Weaknesses:

1. “Toy settings”: Fair point! To address this, we created a realistic same-different dataset using 3D models of objects and used it to evaluate our models, similarly to [1]. See Figure 2 in the supplemental PDF for examples of this dataset. We find that CLIP attains a zero-shot test accuracy of 93.9% on this dataset, while all other models attain chance performance. We also find that CLIP (and DINOv2, despite only achieving chance accuracy) exhibits similar two-stage processing on the realistic stimuli without additional fine-tuning on them (albeit with more attention paid to background tokens). This generalizes our results from the toy setting. See Figure 3 in the supplemental PDF for attention pattern analyses on these stimuli. We note that this task is not a perfect analogue to our synthetic setting due to more realistic perceptual variation (i.e. lighting, rotation, occlusion), but hopefully its inclusion can ameliorate concerns about real-world data/tasks!

2. “Confirmation bias”: This is a reasonable concern. However, we try to let the data speak for themselves. Though our attention pattern analysis might be more qualitative and up for subjective interpretation, our analyses in sections 4 and 5 make clear that there is a marked difference between early and late processing along the lines of what we have described as a “two-stage processing pipeline.”

3. “Clearer annotations”: We completely agree. Upon acceptance, we plan to use the extra space in the final submission to make this figure larger and include more detailed annotations. We are happy to clarify our interpretation of Figure 2 as well (especially in the caption).

4. “Qualitative examples”: We have some qualitative examples in Figure 11 (in the appendix), but we have created a new figure that gives much more intuition and have also written a greatly expanded appendix outlining this and Figure 11. See Figure 4 in the supplemental PDF for the new figure.

5. “Improve model relational reasoning”: In the submission, we attempted to induce relational reasoning by encouraging models to form local, object-level representations using an auxiliary loss in Section 7. However, your point is very well taken, and we had similar thoughts. In light of this, we have devised a new auxiliary loss derived from the attention head scores in Section 3 that explicitly encourages models to exhibit two-stage processing. We hypothesized that this would improve relational reasoning. We find that training randomly-initialized ViTs on the discrimination task with this additional loss term significantly boosts performance (76.5% to 93.9% test accuracy; +17.4). It also boosts the model’s compositional generalization (75.9% to 92.3% accuracy; +16.4). We have promising preliminary results using this loss to improve performance on the RMTS task and will comment on them when ready. This loss is also quite general, and we believe that it can be easily adapted to and possibly improve performance for a wide variety of relations. We will include these new empirical experiments and details about the loss in the main body of the paper upon acceptance.

Questions:

1. “Real world images”: See response to weakness #1.

2. “Logit lens”: At your suggestion, we have implemented and run a logit lens analysis on several models, analyzing how different model components contribute to “same” and “different” classifications. The analysis reveals that CLIP seems to process examples in a qualitatively different way compared to the rest of the models, perhaps pointing towards a mechanism for its success on the realistic stimuli. In particular, CLIP appears to make greater use of register tokens in the background compared to other models; it also appears to use the CLS token in non-intuitive ways for intermediate computations (rather than just gradually storing the “same” or “different” decision in CLS across layers, as other models do). We will include this analysis in the appendix, though we leave further investigation to future work.

3. “Improve relational reasoning”: See response to weakness #5.

4. “Alignment with text descriptions”: In general, a vision model that cannot perform relational reasoning will not discriminate between closely matched sentences (i.e. “plants surrounding a lightbulb” vs. “a lightbulb surrounding plants”). Indeed, this has been born out in the literature in the form of Winoground [2] among other datasets.

Citations:

1. Tartaglini, A. R., Feucht, S., Lepori, M. A., Vong, W. K., Lovering, C., Lake, B. M., & Pavlick, E. (2023). Deep neural networks can learn generalizable same-different visual relations. arXiv preprint arXiv:2310.09612.
2. Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022). Winoground: Probing vision and language models for visio-linguistic compositionality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5238-5248).

Thank you for your thoughtful comments and questions. Find point by point responses below:

Weaknesses:

1. “...Literature review for context…”: We completely agree. Upon acceptance, we will use the extra space to expand our literature review section to include more insights from both the (vast) cognitive science literature and the mechanistic interpretability literature (see global rebuttal #1 for specific topics).

2. “Flow of the paper”: We are happy to add more signposting throughout the sections. Roughly, the paper is structured such that Section 3 motivates the investigation of two stages of processing, Sections 4 and 5 respectively investigate these stages in greater detail, and Sections 6 and 7 discuss the implications of some of our findings on downstream performance. We are happy to state the structure of the paper in the introduction!

3. “Dataset is not well discussed”: We have written a vastly extended appendix detailing the dataset and its construction, which we will include in the final submission. We are also happy to answer any questions about the dataset during the discussion.

4. “CLIP ViTs”: In this work, we investigate abstract relations in a variety of ViTs. In particular, we study ViTs that are pretrained with different kinds of supervision, e.g. CLIP, DINO, and ImageNet. However, as CLIP is the only type of pretraining that achieves near perfect test accuracy for our tasks, we focus on it for the majority of the paper. Since submission, we have added DINOv2 to our analyses, finding that it solves both discrimination and Relational-Match-to-Sample (RMTS) tasks with near perfect test accuracy (99.5% test accuracy on discrimination and 98.2% on RMTS); importantly, it also exhibits two distinct stages of processing similar to those we observe in CLIP. This generalizes our findings beyond CLIP and bolsters our intuition that pretraining data scale (rather than something particular to CLIP) provides the visual representations needed to solve these tasks. We will include these results in a new appendix to the paper and reference them throughout the main body.

Questions:

1. “Little progress”: Great point—we should be more specific. We mean that across a variety of tightly controlled benchmarks, vision models tend to struggle with tasks that involve relations [3, 4], especially compared to tasks that involve semantic computations. Additionally, the models that you mention have demonstrated some progress in processing visual relations, but there are still shortcomings to be addressed. We are happy to add these references to the final manuscript in an updated related work section.

2. “Algorithms”: This terminology is borrowed partially from the mechanistic interpretability literature (i.e. in conceptualizing neural networks as implementing interpretable algorithms), and partially from the cognitive science literature (i.e. Marr’s levels of analysis). When we use the word “algorithm”, we are referring to the series of representations generated by a model to solve a particular task.

3. “Existing datasets”: There are plenty of datasets used to study visual relations! For example: SVRT [1] and CVRT [5]. However, these datasets are not controlled enough to enable us to adapt state-of-the-art techniques from language model mechanistic interpretability for ViTs. Our dataset is explicitly constructed to enable these techniques by e.g. consistently aligning objects within the bounds of ViT patches. Our new dataset appendix details our design choices.

4. “Motivation of using CLIP ViT”: As mentioned above (weakness #4), we use many different ViTs. Your point about CNNs being more human-like is well taken! We use ViTs because prior work has demonstrated that they can solve same-different tasks in a robust fashion [2]; they also enable us to use techniques from mechanistic interpretability that have previously been developed for Transformer language models (DAS, attention analysis, linear probing-based causal interventions, logit lens). Since CNNs lack tokens, it is not clear whether it is possible to apply these techniques to them.

5. “Disentangled model for generalization”: We are happy to include this reference in the main body of the paper. Notably, our work demonstrates the benefits of disentangled representations for compositional generalization (rather than standard generalization) in a vastly different setting!

6. “Build a more robust version”: We were very motivated by this question as well! Since submission, we have implemented a new auxiliary loss function derived from the attention head scores in Section 3 that encourages the model to adopt two-stage processing. We find that this helps a randomly-initialized ViT achieve significantly better downstream performance on the discrimination task (76.5% to 93.9% test accuracy; +17.4). It also significantly boosts compositional generalization accuracy for discrimination (75.9% to 92.3% accuracy; +16.4). We currently have promising preliminary results using this loss to improve performance on the RMTS task and will comment on them when they are ready. This loss is also very general and could possibly be used in the future to improve relational reasoning for a wide variety of visual relations. We will include these results and more details about the loss in the main body of the paper upon submission.

Citations:

1. Fleuret, F., et al. (2011). Comparing machines and humans on a visual categorization test.
2. Tartaglini, A. R., et al. (2023). Deep neural networks can learn generalizable same-different visual relations.
3. Thrush, T. et al. (2022). Winoground: Probing vision and language models for visio-linguistic compositionality.
4. Zeng, Y., et al. (2024). Investigating Compositional Challenges in Vision-Language Models for Visual Grounding.
5. Zerroug, A., et al. (2022). A benchmark for compositional visual reasoning.

Thank you for your thoughtful review! Find point-by-point responses below:

Weaknesses:

1. “Simple relations”: This is fair! With respect to “higher order relations”, we attempted to explore this using the Relational-Match-to-Sample (RMTS) task—an explicitly hierarchical version of the discrimination task. While this goes beyond much of the same-different literature (which focuses on variations of the discrimination task), we do limit our investigation to the same-different relation (rather than other relations) due to its particular significance in the study of abstract concepts in the cognitive sciences, as well as its conceptual simplicity for applying mechanistic interpretability techniques to.

2. “Detailed intuition”: We agree that this description (pasted below) was somewhat speculative during the submission:

“Raghu et al. (2021) finds that models pretrained on more data tend to learn local attention patterns in early layers, followed by global patterns in later layers. This might give CLIP a particular incentive to create local object representations, which are then used in relational operations. Future work might test this hypothesis.”

• Since submission, we have rerun our behavioral analysis and attention pattern analysis on a DINOv2 ViT, which is pretrained on about 142M images; this is close in size (in order of magnitude) to CLIP’s pretraining dataset (about 400M images). Despite having a vastly different pretraining objective to CLIP, the DINOv2 model matches CLIP’s performance on both discrimination and RMTS tasks (99.5% test accuracy on discrimination and 98.2% on RMTS). It also exhibits two-stage processing like CLIP. This bolsters our intuition that pretraining data scale (rather than the type of pretraining supervision) is the key. We will include all of these results in the camera ready version. See Figure 1 in the supplemental PDF for supporting figures.

Questions:

1. “WO attention score is higher than WP”: These attention scores refer to the maximum proportion of attention paid by any given attention head to either within-object, within-pair, between-pair, or background tokens. Though the max for WO is higher than the max for WP in later layers, it is more informative to look at the trend within a given head-type rather than between head types. Notably, we find that the peaks of these scores occur in the expected sequence.

2. “Analysis of some models are missing”: Thank you for catching this! We will include these graphs in the appendix of the final submission.

3. “MAE”: Can do! We fine-tune a pretrained ViT-MAE model on the discrimination and RMTS tasks and perform an attention pattern analysis (following Section 3). We find that ViT-MAE achieves very similar performance to ImageNet and DINO ViT. On discrimination, it achieves 98% test accuracy and 94.9% compositional generalization accuracy. On RMTS, it achieves 93.4% test accuracy (interestingly, somewhat higher than ImageNet and DINO) and 85.3% compositional generalization accuracy. Its attention patterns do not demonstrate two-stage processing like CLIP or DINOv2; instead, the local and global heads are mixed throughout the layers. We will add an attention pattern analysis on MAE to the appendix of the paper.

4. “Figure 6”: No—each data point corresponds to a different model, which achieves a different maximum disentanglement score. We will revise the caption for clarity.

Citations:

1. Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021). Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34, 12116-12128.

We have continued to experiment with integrating attention-based losses to improve model performance in the RMTS task. We have found that combining an attention based auxiliary loss + the shape/color loss described in the manuscript improves performance dramatically for a randomly-initialized model trained on the RMTS task. Specifically, ViT B/16 accuracy improves from ~50% train, test, and compositional generalization accuracy without auxiliary losses to 95% train accuracy (+45%) , 91% validation accuracy (+41%) , and 77.4% compositional generalization accuracy (+27%) with both attention and shape/color losses.

Notably, merely including the shape/color auxiliary loss results in far weaker performance improvements, at 54% train, test, and compositional generalization accuracy. These results are already present in Table 4 of the manuscript.

In summary, this experiment further supports our main conclusion: that both multi-stage relational processing and well-structured object representations are required to solve the RMTS task. Inducing both of these factors is possible using straightforward auxiliary losses during training.