Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models

Thank you for your thoughtful and constructive review! We appreciate your recognition that this is a “creative way to [...] isolate configural shape sensitivity, controlling for texture completely”. Below, we address specific comments.

---

“CSS is strongly correlated with other shape‑sensitive metrics, so the insight feels incremental.” and “What specific types of failure modes or model differences would CSS detect that previous methods would miss?”

We respectfully disagree with the reviewer that the correlation with other shape metrics makes our contribution incremental. The shape-vs-texture bias measure is relative and thus can give a false sense of successful shape sensitivity. For example, shape-vs-texture bias can rise simply by suppressing texture (e.g., as in Stylised, Adversarial, Top-k CNNs), while their CSS hardly changes (Fig. 2C). In other words they are not more sensitive to holistic shape (which the CSS measure tells you clearly and directly). Moreover, CSS explains variance on four downstream, shape-dependent benchmarks far better than shape-vs-texture bias (Sec. 9, Williams test), revealing failure modes that earlier diagnostics would have missed. In other words, we argue that CSS is a more targeted assay of the underlying construct of interest, which is about how models leverage holistic shape processing (beyond local cues). We will clarify this measure's advantage over existing ones in the Camera-Ready version.

---

“While MAEs are included in the CSS benchmark and perform moderately, the paper does not explore why they underperform compared to models like DINOv2.”

You are correct and we agree that the moderate CSS performance of MAEs merits further explanation (with the page limit we had to make some choice). To address your concern we ran the representational similarity analysis on Hiera-MAEs and two other masked-image-models in our model suite - BEiT (token prediction) and BEiT-v2 (token prediction with CLIP-distilled targets), which share MAE’s mask-and-reconstruct recipe and differ in the semantic level of the prediction target - target either being 8096-D code-book indices from a vector-quantised VAE that was itself trained (without labels) to reconstruct pixels in BEiT or soft probability vectors distilled from a CLIP vision encoder. We find that only BEiT-v2 develops the mid-layer switch from part-based to category-based representation that we had observed in other high-CSS models (see Fig. 4B). The evidence suggests that just masked modeling (both, pixel or latent level reconstruction) is not sufficient in a purely self-supervised setting, instead DINO’s multi-view contrastive loss that enforces global consistency across large spatial crops appears to be critical to unclocking this configural capacity. When such global pressure is absent, comparable CSS can still be reached by injecting language-aligned targets, as seen in EVA-CLIP and BEiT-v2. Thus, not all self-supervision is equal.

We did not perform attention ablation studies on these MAEs, because probing them with the same radius-mask ablation used for global-attention ViTs is technically challenging without substantial redesign: Hiera-MAE starts with a 7 × 7 conv stride-4, then uses stage-wise local windows that shrink the receptive grid (56 → 28 → 14 → 7). A fixed “mask radius = k tokens” therefore corresponds to four different pixel extents across stages, making a layer-wise comparison impossible without redefining distance at each stage. In stages 0–2 each head already attends only within 8 × 8 or 4 × 4 windows, additional masking outside removes information the model never sees, probing a different phenomenon than our global-attention test. For clarity, we limited Fig. 4 to uniform-grid ViTs (DINO-B/14, ViT-B/16).

We appreciate this suggestion, which helped clarify which self-supervised objectives drive configural sensitivity. We will include this point in the camera-ready version.

---

“High CSS might just reflect heavy spatial augmentations like random cropping in DINOv2.”

Fair point. We think heavy spatial augmentations alone does not explain high CSS. Our model suite contains models trained with heavy augmentations like RandAugment (Cubuk et al., 2019), Mixup (Zhang et al., 2018) and Cutmix (Yun et al., 2019):

1. Multiple variants of ResNets from Wightman et al., 2021. These models are trained with the A1 augmentation recipe with RandomResizedCrop (scale 0.08–1.0), RandAugment, Mixup/CutMix, color-jitter, and flips.

2. ViT-S/16 AugReg and ViT-B/16 AugReg from Steiner et al., 2021 which applies RandomResizedCrop + RandAugment + Mixup augmentations.

These augmentations however are still fed to the network independently under a cross-entropy loss, no link is made between different views of the same object. The CSS scores for these models reach up to 36.11%. DINO, by contrast, pairs two spatially distant crops of the same image (local and global view) and forces their embeddings to coincide, so the model is directly rewarded for integrating long-range information that spans the global shape of the object in the image. So rather than random cropping which is also present in the DINO framework, we think that the high CSS scores for DINO models can be attributed to its objective that enforces a global-consistency constraint.

---

“Transformer models may benefit unfairly from explicit positional encodings matched to the anagram grid.” and “Have you compared models with different position encodings?”

Really good point, and we also thought so! We evaluated five new additional variants of the ViT-B/16 architecture from timm, each with a different positional encoding scheme. For context, the standard supervised ViT-B/16 from our paper (which uses learned absolute positional embeddings) has a CSS of 23.61%.

New Experimental Results:

• ViT with Relative Positional Embeddings: 38.89% CSS
• ViT with RoPE (Rotary Position Embedding): 51.38% CSS
• ViT with RoPE (Rotary Position Embedding) + Absolute PE: 51.38% CSS
• ViT with Mixed RoPE: 48.61% CSS
• ViT with Mixed RoPE + Absolute PE: 52.77% CSS

The stronger performance of models with relative position encodings (like RoPE) demonstrates that CSS does reward models for processing relational spatial structure. We will add this new analysis to the appendix to provide a more complete picture of the role of positional encodings. Thank you for this valuable suggestion.

---

“Can you clarify whether the representational similarity analyses provide insight beyond the expected hierarchical abstraction of deep networks?"

We agree a simple transition from low-level to task-driven features is expected in deep networks. However, our analysis provides insight beyond this as we found qualitative difference in ‘how’ models make this transition, specifically two distinct patterns:

1. The "Stuck-on-Local" Pattern: In low-CSS models, the representations remain dominated by local patch similarity across all layers. For the anagram pair (same patches, different objects), the final representations remain highly similar: the model cannot pull them apart. These models fail to make the full transition to a global representation when confronted with matching local evidence.

2. The "Configural Flip" Pattern: In high-CSS models, we see a non-trivial "representational flip". While early layers are sensitive to the shared patches, the intermediate layers actively work to overcome this. By the final layers, the representations of the two different objects in an anagram pair are just as dissimilar as if they were built from entirely different patches. The model’s representation becomes fully determined by the global configuration, successfully overriding the misleading matched local cues.

Therefore, the insight is not simply that a hierarchy exists. The novel insight is the existence of this "configural flip" as a hallmark of high-performing models and its absence in weaker ones. These high-CSS models are not just going from a local to a task-driven representation, but instead, have an intermediate transition of local enrichment due to long-range contextual interaction which ends up reflecting as the “configural flip” signature. This analysis provides a way to distinguish these models from those whose final output is still contaminated by local feature similarity.

---

“Can you evaluate CNN-based self-supervised models such as MoCo, SimCLR, BYOL, or SwAV to determine whether high CSS requires attention, or if it can arise from training objectives and augmentations alone?”

We could not include MoCo, SimCLR, BYOL, or SwAV, as pretrained versions with 1000-way classification heads were not available. However, our suite includes SSL-ResNet-50 and SWSL-ResNet-50, which offer partial coverage. We agree this is a valuable direction and plan to test it systematically by holding architecture and data constant while varying the self-supervised objective.

---

“Does the 64×64 patch size of the anagrams mean CSS mostly measures layout rather than sub‑patch integration?”

Of course CSS requires both subpatch processing and global “layout” sensitivity, but it specifically targets the latter. Since both images in an anagram pair share identical parts, models must rely on configural information to succeed. This is why BagNets, which lack long-range integration, perform at chance.

---

“Did you try training or fine-tuning DINOv2 (or a similar ViT) without spatial augmentations [...] to assess whether positional robustness is necessary for high CSS?”

We did not retrain DINOv2 without multi-crop augmentations, as this would require full pretraining from scratch on large-scale data, which is beyond the scope of this paper. Our goal is to evaluate emergent properties of existing models. That said, we agree this is an important question, and a controlled comparison isolating the role of augmentations would be a valuable direction for future work.

Thank you for your positive feedback on the quality and significance of our work. We truly appreciate your statement that “this dataset and CSS can/should be utilized by many vision researchers in the future,” and we wholeheartedly agree with this perspective.

---

“The abstract can be substantially improved by making it more succinct.”

Yes, we agree. We'll revise the abstract to make it more succinct and group the key findings more clearly. We will edit the abstract to be shorter and more clear in the camera ready version.

---

“The experiments are extensive, although I do agree with the authors that they could have (perhaps marginally) benefited from creating a larger dataset.”

You're absolutely right, and we agree that a larger dataset strengthens the findings. We've already expanded the corpus to 1440 visual anagram pairs, a 20x increase, and re-evaluated all models. The results are highly consistent: model scores remain stable, and the relative CSS rankings of the models are preserved, the correlation between the CSS scores on the original and expanded datasets is r = 0.99. These updated results and the full dataset will be included in the camera-ready version.

---

“How many human participants were recruited for the study? This should be clearly stated.”

Only two participants were tested, as the goal of this study was only to confirm that the images in anagram pairs are easily recognizable. These images are not meant to to challenge human recognition, and the small sample already verify this. Moreover, the results are not intended for making direct comparisons between model and human behavior, which would require a larger-N (Alvarez & Konkle, 2024). We will clarify the role of these behavioral data in the manuscript.

---

Have the authors plotted “Top1 ImageNet Accuracy” vs. “Shape-vs-Texture Bias”? I believe a plot like this could be interesting if it is shown next to the “Top1 ImageNet Accuracy” vs. “CSS” plot in Figure 2. Would the R^2 be lower for Shape-vs-Texture Bias?

We found that adding an extra subplot to the Figure 2 disrupted the clarity a bit, but we instead will put a new figure in the Appendix, which also will show all pairwise relationships among CSS, Shape-vs-Texture Bias, Top-1 Accuracy, and related metrics. Thank you for your suggestion.

We thank you for the generous evaluation and sharp comments. We particularly appreciate the recognition that the study is “clever and well‑motivated,” features “lucid writing and clear figures,” and delivers “a wonderful paper” that could be “a contender for a spotlight talk,” pending minor clarifications. Below, we address specific comments.

---

“There seems to be a relatively small number of image categories (nine) generated. Also how many image pairs were generated in total? I couldn't find an exact number.”

We agree with the reviewer on the small number of anagrams. The benchmark initially contained 72 total visual anagram pairs, where the choice of the specific 9 categories was motivated by Baker & Elder, iScience, 2022. We have now expanded the number of total stimuli by 20x, from 72 to 1440 visual‑anagram pairs. We re-evaluated all models on this new, larger dataset. Preliminary results show that the scores are very stable and the relative CSS rankings of the models are preserved, the correlation between the CSS scores on the original and expanded datasets is r = 0.99. We will include the expanded set and report these results in the camera-ready version of the paper!

---

“Figure 3 shows a suggestive comparison of two networks, but are the trends here consistent across the broader range of transformer architectures? What about transformer architectures that are trained with smaller context windows (rather than ablated afterwards)?”

Fair point. Our original goal was to provide a clear, controlled comparison between a high-CSS and a low-CSS model with similar ImageNet performance, which is why we focused on DINOv2-B/14 and ViT-B/16. However, since then, we expanded this analysis to the remaining DINO models ( DINOv2-S/14, DINOv2-L/14, DINOv2-G/14). When applying the same ablation paradigm, the effects in other models were less pronounced, particularly the deeper models. We hypothesized the single-block ablation may be too subtle. To address this limitation, we designed a cumulative ablation experiment: First, we ablated all blocks up to block ‘n’ to isolate the contribution of the network up to that point and second we ablated all blocks after block ‘n’ to see if there is recovery by later layers. This analysis showed that the drop is the highest and most irrecoverable (even for the DINO large and giant variants) in the intermediate layers. We will include this cumulative ablation analysis in the camera ready version. While this set is not the full range of transformer architectures, we hope the expanded analysis helps provide more support for the trends we observed. Thank you for this question.

---

“All of the visual anagrams seem to be prompted to have "black paint texture". While this controls that the two images have the same texture, it may mean the texture is out-of-distribution. This complicates my intuition and interpretation of the results (see question below).”

Indeed, we intentionally used a homogeneous "black paint texture" to control and match the local statistics, as this helps further ensure that any classification difference would stem from the global configuration. However, it’s a fair question to wonder if this unnatural texture is enough out-of-distribution (OOD) that holistic mechanisms observed here do not apply to more ‘natural’ images. To make the concern more concrete, is poor CSS performance actually due to a failure of OOD texture, rather than a failure of configural processing?

There are several results that address this concern. The main evidence comes from single-image accuracy for the anagram images – if the texture itself were the problem, models should fail to classify even individual anagram images. This is not the case. For example, a ResNet-50 classifies single anagram images with 52.7% accuracy (well above the 11% chance rate). Its CSS, which requires correctly identifying both images in a pair, plummets to 16.7%. Similarly Bagnet 33 is at 5% on CSS but at 39% for individual images. This shows the model can extract meaningful features from one image, but fails when it must distinguish two configurations built from the exact same local features.

Further, we also know that poor CSS models are accurately representing the local texture: the representational similarity analysis shows that these models show high correlations between same-part pairs (Figure 4, green lines), and low similarity between different-part pairs (blue-and orange) indicating they have strong representations of the local details. If all the “black paint textures” were treated the same on account of being out of distribution, none of these results would hold (all correlations would be high, and all three lines would be overlapping).

That said, we agree with you that employing a richer set of matched textures would improve ecological validity, and we will highlight this as an important direction for future work in our discussion.

---

You keep using the world "multiset" to describe the set of patches that the visual anagrams share. How is this a multiset? Can patches occur more than once in the set?

Apologies for the confusion. We will change this word to “set” in the places it occurs in the manuscript. (While we were initially trying to use it to clarify the difference between unordered and spatially ordered patches, we realized the sentences in the paper are actually clearer and still accurate when we use the word “set”).

---

"Are rotations of the patches allowed when generating the image…"

Yes. The transformation involves rotation of the patches. We will update Section 3 in the camera-ready version to explicitly state that the transformation involves both permutation and rotation, which provides the necessary flexibility to form two distinct and recognizable object shapes from the same collection of local parts.

---

You say that this work offers "actionable design principles for future vision systems". Can you be more explicit about what these design principles are?

For sure, thank you for this question! The core actionable principles derived from our findings are:

1. Prioritize Training Objectives that Enforce Global Structure. Our results, consistent with prior findings, also show that models trained with standard supervised classification often solve the task using local texture shortcuts. In contrast, self-supervised objectives that require matching global and local views (like in DINOv2) or multimodal objectives that align vision with language (like in EVA-CLIP and SigLIP2) are far more effective at inducing robust holistic processing. Choosing the right objective is critical.

2. Ensure Architectures Can Perform Long-Range Spatial Integration. Our mechanistic analyses (Sec. 6 & 7) show that the ability to relate distant image patches in intermediate layers is the key computational mechanism behind high CSS performance. The complete failure of architecturally-local models like BagNet reinforces this: without the capacity for long-range communication, models cannot develop configural understanding. It is an open question if post- and pre-training methods that amplify these mechanisms in standard ViTs would unlock such a capacity without needing to train on large orders of data.

3. Replace shape-vs-texture measure with an explicit shape measure. Our work shows that improving relative shape-vs-texture bias does not guarantee better configural processing. The principle is that these are distinct abilities. Rather than simply discouraging texture use, future work could also directly measure and optimize for absolute configural competence.

We will add a dedicated paragraph in our camera-ready version to clearly outline these principles for future model development.

---

“(Minor) The authors use the term "representational similarity analysis" to describe their analysis in Figure 4. This is confusing [...] To avoid confusion, I suggest the authors come up with different language”

We see that this terminology is unfamiliar, and we modified the text to more carefully introduce the term. This terminology was originally introduced in the computational neuroscience community (Kriegeskorte, 2008), and is increasingly being adopted in the NeuroAI community (e.g., “Getting aligned on representational alignment”, Sucholutsky 2023), and in ML research focused on similarity of neural network representations (Kornblith et al., 2019).

---

Minor typos

Thank you so much for pointing these out. We will correct it in the revised draft.

Thank you for the clear summary, the nice comments and targeted suggestions. We particularly appreciate your acknowledgement that the paper “presents a very interesting metric for configural processing. Below we address each of your points in turn.

---

“It would be good to have more examples (currently 72 pairs) in the dataset to get a more robust score.”

We agree that a larger dataset is crucial for ensuring the robustness of our findings. In response, we have increased the corpus to 1440 visual‑anagram pairs ( a 20x increase in dataset size). We re-evaluated all models on this new, larger dataset. Preliminary results show that the scores for models are very stable and the relative CSS rankings of the models are preserved, the correlation between the CSS scores on the original and expanded datasets is r = 0.99. We will include the expanded set and report these results in the camera-ready version of the paper!

---

“Why not do the attentional ablation study in section 6 for all or a few more (attenion-based) models? It might be interesting to see if these results hold for other models as well.”

We now expand this analysis to the remaining DINO models ( DINOv2-S/14, DINOv2-L/14, DINOv2-G/14). When applying the same ablation paradigm, the effects in other models were less pronounced, particularly the deeper models. We hypothesized the single-block ablation may be too subtle. To address this limitation, we designed a cumulative ablation experiment: First, we ablated all blocks up to block ‘n’ to isolate the contribution of the network up to that point and second we ablated all blocks after block ‘n’ to see if there is recovery by later layers. This analysis showed that, across all DINO models tested, the drop is the highest and most irrecoverable (even for the DINO large and giant variants) in the intermediate layers. We will include this cumulative ablation analysis in the camera ready version. Thank you for this suggestion.

---

“More discussion of the results, especially why certain models have higher CSS, would be helpful. Why do the authors think self-supervised models do better than supervised ones for example?”

We were also genuinely wondering about this and we propose that two main factors contribute to high-CSS scores: The first is an objective that includes some form of global/local contrastive loss. For example the high-CSS DinoV2 models include a direct contrastive loss between global views (large crops), and local views (small crops) from the same image. Other high CSS models such as CLIP, SigLIP, SigLIP2, Beit2, Eva-CLIP are directly or indirectly language-aligned. For instance, EVA-CLIP uses masked image modeling but is distilled from a CLIP-trained vision encoder while BEiT-2 is trained to predict masked discrete visual tokens from a semantic CLIP tokenizer. To this end, language, either through direct contrastive learning or via knowledge distillation, provides a “global view” of the image. The pressure for local representations to align with global representations seems to be a key factor giving rise to high-CSS scores, and the ViT architecture (with global attention) seems particularly well-suited to developing high-CSS.

The second possible factor is training on a large, diverse dataset. For example, it seems that the high CSS models were also trained on larger datasets. So, disentangling what CSS benefits are confirmed by the training loss vs the dataset (or an interaction of the two, where benefits of the training objective work best with global attention and at a certain dataset scale and diversity) is important work to be done. We will expand our discussion of these results and our hypotheses about why self-supervised models seem to do better than supervised ones in the camera ready version.

---

“It would be good to have a full list of all models in the appendix.”

Absolutely, we will add a detailed list in the appendix for all the evaluated models.

---

“When the images are fed to ViT based models, are you aligning the ViT grid with the anagram's grid? What happens if it is not aligned and one token in ViT actually straddles multiple patches in the anagram? I don't think this will make much of a difference but I was just curious.”

The pieces of the anagram are not actually aligned with the VIT grid! In fact, the anagram pieces are actually shaped like puzzle pieces (not perfect squares, as was indicated in the caption–we have fixed this)--this means there is no way to align these anagram pieces within the ViT grid. Further, the anagram patches are from a coarse 4x4 grid of 64x64 px pieces, while the ViT16 patches are 16 x16px patches; thus, there are approximately 4 VIT patches per anagram puzzle piece. (To address worries about “border hacking”, in case new local features are created at the interface between puzzle pieces that could account for the performance instead of holistic processes, we have section 8 using bag net models to rule out this possibility.) Thank you for asking these questions; we will clarify these details.

---

Summary: We think with the 20x expansion of the number of anagrams images, and the extended ablation analyses, the paper will be much improved. Thanks for your feedback.