Response to reviewer 4Wjo (1/1)

Dear Reviewer 4Wjo,

We thank the reviewer for their careful reading and for recognizing the significance of our research question and robust quantitative results. We appreciate your constructive feedback and are delighted that we were able to run more experiments (across different objectives) and that they substantially improved our paper.

---

[W1, Q1] Limited verification across pretraining objectives

We have now extended our analysis beyond DINOv2 to include CLIP, MAE, and supervised-only trained ViTs (Some limited experiments with CLIP and DINO model sizes were already included in the supplementary: Lines 471-478, Appendix Figure 7). To ensure fair comparisons, we now standardized the effective patch coverage by resizing inputs so that all models process the same spatial patch divisions. This ensures a shared baseline (72.6%) across all models (corresponding to the accuracy of always predicting "different," which accounts for the class imbalance in the dataset).

Here are our results:

Table: Probe accuracy on the IsSameObject task across ViTs

This now allows us to produce much more interesting insights: CLIP, MAE, and DINO all demonstrate object binding capabilities, supporting our broader claims about Large Pretrained Vision Transformers beyond DINOv2 alone. We also show that supervised ImageNet classification yields poor object‑binding performance, indicating that binding is an acquired ability under specific pretraining objectives rather than being a universal property of all vision models.

We offer explanations of why these training regimes work:

• DINO: As noted in Lines 186–189, the DINO loss drives the model toward object-level features by enforcing consistency between augmented views, since each view contains the same objects and the model must align their representations; this objective has been shown to encourage class and object specific features (Caron et al. ICCV 2021), as the reviewer points out.
• CLIP: By aligning images with text captions, CLIP effectively assigns each object a symbolic label (e.g., “the cat”), which can act like a pointer that pulls together all patches of that object. This supervision likely encourages patches from the same object to cluster in feature space. Future work could track object‑binding strength across CLIP training checkpoints to validate this hypothesis.
• MAE: The masked autoencoder objective requires the model to reconstruct a missing patch from its surroundings. When the masked patch sits between multiple objects, correctly predicting its content forces the model to infer which object it belongs to, thus promoting the grouping of patches from the same object.

Our findings thus produce a much wider coverage of ViTs and we provide an understanding of potential reasons for why binding emerges.

---

[W2] Probe failure with multiple same‑class objects

We agree that object-class and object-identity probes could fail if there are multiple objects from the same class. We’d like to clarify that these probes are used only to test possible hypotheses about how binding works in ViTs: in this case, that the model determines “same‑objectness” by identifying each token’s class or identity and then comparing the token pair.

To evaluate this, we compared probe accuracies across designs: the quadratic probe consistently outperforms both object‑class and object‑identity probes in most layers (Figure 3), indicating that DINOv2‑Large employs a more sophisticated binding strategy than simply classifying tokens by object class or identity and comparing them.

[W3] Limited generalizability shown in Figure 4

The reviewer questions whether Figure 4 consistently supports our claim of emergent object binding. We’d like to point out that Figure 4 is intended as a stress test, which shows extreme cases with identical objects or same‑class objects in one image to highlight where ViTs may struggle (e.g., distinguishing identical instances or class mates). It is not meant to represent average performance; Figure 3 provides the overall evaluation of object binding across layers and models.

We also note that IsSameObject performance degrades at the final layer (23), likely because DINO’s self‑distillation loss trains only the [CLS] token’s projection‑head output, leaving last‑layer patch‑token activations without direct supervision and prone to drift.

---

[W4] Ambiguous calculation of IsSameObject scores and correlations

IsSameObject scores are the 0–1 outputs of our trained quadratic probe, reflecting the model’s assessed probability that two patches belong to the same object. Attention weights are detailed in Supplementary Lines 530–538. We compute the correlation as the Pearson correlation between all token‑pair IsSameObject scores and their corresponding attention weights.

---

[Q] Ground-truth label clarification

We’d like to clarify that we always use instance‑level ground truth: different objects, even when they belong to the same class, are always labeled as “different” during probe training and evaluation. Therefore, the confusion seen in Figure 4 is not due to coarse labels but reflects genuine challenges in the model’s binding ability.

Figure 4 evaluates object binding under extreme conditions where multiple identical or same‑class instances in a single image. It shows that, while ViTs do exhibit object binding, it is not flawless: identical objects can still be confused.

---

Minor concerns

We will correct the listed typos and enhance both our methods' presentation and the rigor of our figures.

We believe that our responses have dramatically strengthened the contributions of our work and are thankful for the reviewer's feedback.

Best regards,
Authors

Response to reviewer gvWE (1/1)

Dear Reviewer gvWE,

We thank the reviewer for their careful reading and for recognizing our novel IsSameObject hypothesis, probe design, and robust quantitative results. We appreciate your feedback and are delighted that we were able to run experiments across different objectives and that they substantially improved our paper.

---

[W1, Q1] Limited verification across pretraining objectives

We have now extended our analysis beyond DINOv2 to include CLIP, MAE, and supervised-only trained ViTs (Some limited experiments with CLIP and DINO model sizes were already included in the supplementary: Lines 471-478, Appendix Figure 7). To ensure fair comparisons, we now standardized the effective patch coverage by resizing inputs so that all models process the same spatial patch divisions. This ensures a shared baseline (72.6%) across all models (corresponding to the accuracy of always predicting "different," which accounts for the class imbalance in the dataset).

Here are our results:

Table: Probe accuracy on the IsSameObject task across ViTs

This now allows us to produce much more interesting insights: CLIP, MAE, and DINO all demonstrate object binding capabilities, supporting our broader claims about Large Pretrained Vision Transformers beyond DINOv2 alone. We also show that supervised ImageNet classification yields poor object‑binding performance, indicating that binding is an acquired ability under specific pretraining objectives rather than being a universal property of all vision models.

We offer explanations of why these training regimes work:

• DINO: As noted in Lines 186–189, the DINO loss drives the model toward object-level features by enforcing consistency between augmented views, since each view contains the same objects and the model must align their representations; this objective has been shown to encourage class and object specific features (Caron et al. ICCV 2021), as the reviewer points out.
• CLIP: By aligning images with text captions, CLIP effectively assigns each object a symbolic label (e.g., “the cat”), which can act like a pointer that pulls together all patches of that object. This supervision likely encourages patches from the same object to cluster in feature space. Future work could track object‑binding strength across CLIP training checkpoints to validate this hypothesis.
• MAE: The masked autoencoder objective requires the model to reconstruct a missing patch from its surroundings. When the masked patch sits between multiple objects, correctly predicting its content forces the model to infer which object it belongs to, thus promoting the grouping of patches from the same object.

Our findings thus produce a much wider coverage of ViTs and we provide an understanding of potential reasons for why binding emerges.

---

[W2, Q3] Limited visual support

Although we cannot include visualizations at this stage, we have validated our claim that mid‑layers capture local object structure and higher layers emphasize object‑class grouping by training new probes. The IsSameObjectClass probe’s accuracy increases in deeper layers as IsSameObject accuracy declines, directly supporting this shift. IsSameObjectClass accuracy drops at the final layer (23), likely because DINO’s self‑distillation loss targets only the [CLS] token’s projection‑head output—meaning the last‑layer patch‑token activations receive no direct training signal and can drift away from class‑level structure.

We also introduced IsSameGreyscale probe for low‑level perceptual features; its accuracy decreases rapidly with depth, suggesting a rapid shift from low-level perceptual features to high-level semantics.

Table: Mean probe accuracy (%) across layers for IsSameObject, IsSameObjectClass, and IsSameGreyscale.

---

[W3, Q2] Implications of low-dimensional IsSameObject space

A low‑dimensional IsSameObject subspace indicates that the binding information is concentrated in just a few directions. This makes learning of those directions which matters for downstream tasks and attention far easier. In this low-dimensional space, we also find that there are large margins between the clusters (Lines 231-237, Figure 5), which makes them easy to separate and guarantees generalization. This type of representation not only supports strong generalization (because large margins between clusters imply robustness to new inputs), but also enables efficient compression: by projecting full token embeddings onto the top binding dimensions, we can distill the binding signal into a much smaller code.

We believe that our responses have dramatically strengthened the contributions of our work and are thankful for the reviewer's feedback.

Best regards,
Authors

Response to reviewer x2fh (1/1)

Dear Reviewer x2fh,

We thank the reviewer for your careful reading, constructive feedback, and positive comments on our contextualization and emphasis of the research problem’s significance. We are delighted that we were able to run the requested experiments (across different objectives) and that they substantially improved our paper.

Below, we address your comments point by point.

---

[W1, Q1] Limited verification across pretraining objectives

We have now extended our analysis beyond DINOv2 to include CLIP, MAE, and supervised-only trained ViTs (Some limited experiments with CLIP and DINO model sizes were already included in the supplementary: Lines 471-478, Appendix Figure 7). To ensure fair comparisons, we now standardized the effective patch coverage by resizing inputs so that all models process the same spatial patch divisions. This ensures a shared baseline (72.6%) across all models (corresponding to the accuracy of always predicting "different," which accounts for the class imbalance in the dataset).

Here are our results:

Table: Probe accuracy on the IsSameObject task across ViTs

This now allows us to produce much more interesting insights: CLIP, MAE, and DINO all demonstrate object binding capabilities, supporting our broader claims about Large Pretrained Vision Transformers beyond DINOv2 alone. We also show that supervised ImageNet classification yields poor object‑binding performance, indicating that binding is an acquired ability under specific pretraining objectives rather than being a universal property of all vision models.

We offer explanations of why these training regimes work:

• DINO: As noted in Lines 186–189, the DINO loss drives the model toward object-level features by enforcing consistency between augmented views, since each view contains the same objects and the model must align their representations; this objective has been shown to encourage class and object specific features (Caron et al. ICCV 2021), as the reviewer points out.
• CLIP: By aligning images with text captions, CLIP effectively assigns each object a symbolic label (e.g., “the cat”), which can act like a pointer that pulls together all patches of that object. This supervision likely encourages patches from the same object to cluster in feature space. Future work could track object‑binding strength across CLIP training checkpoints to validate this hypothesis.
• MAE: The masked autoencoder objective requires the model to reconstruct a missing patch from its surroundings. When the masked patch sits between multiple objects, correctly predicting its content forces the model to infer which object it belongs to, thus promoting the grouping of patches from the same object.

Our findings thus produce a much wider coverage of ViTs and we provide an understanding of potential reasons for why binding emerges.

---

[W2, Q2] Cross-layer binding

We thank the reviewer for highlighting cross‑layer interactions as a novel and necessary perspective on studying binding in ViTs; since brains bind across layers, we ask whether ViTs do too.

Cross-layer binding may actively happen under these conditions:

• Layer-specific information processing: ViTs represent different types of information at different layers. Although the self-attention implements a "full range" interaction (all tokens to all tokens interaction), the model learns to limit the "interaction" between the correct tokens and integrate features of the object they belong to as we go deeper.
• Information retrieval from earlier layers: ViTs may access information from earlier layers when current layers lack necessary binding cues. As shown in Appendix Figure 11 and in Amir et al. ECCVW 2022, positional information is gradually removed as the network deepens, and other low-level features may also be discarded. This loss may lead the model to retrieve binding cues from earlier layers instead of keeping them redundantly; evaluating this idea needs further work.

Cross-layer interactions in ViTs can occur via the residual skip connections, so we also train quadratic probes between non-adjacent layers. Specifically, we compute $\phi(x,y) = x^TW_1^T W_2 y$, where $W_{1}$ and $W_{2}$ are the learned projection matrices from layer 15 and layer 18, respectively. The probe accuracies are:

• Layer 15: 89.0%
• Layer 18: 90.1%
• Cross (15→18): 83.3%

The cross-layer probe shows decent cross-layer object binding, and a certain degree of object binding across the system.

---

[W3, Q3] Consensus on ViT object‑binding limitations

We agree that our claim was not well articulated and could lead to misunderstanding. We should clarify that the cognitive- and neuro-scientists believe that binding requires a set of interacting mechanisms such as memory and recurrent processing trained on complex behavior (please see Roelfsema 2023, Solving the binding problem; Peters et al. 2021, Capturing the objects of vision with neural networks; Salehi et al. 2024, Modeling Attention and Binding in the Brain through Bidirectional Recurrent Gating). Therefore, our claim mentioned ViTs as part of the general set of artificial neural networks that do not meet these hypothetical requirements. Here we should emphasize that this hypothesis is now questioned by some cognitive neuroscientists (Scholte et al. 2025, Beyond binding: from modular to natural vision).

We would like to point out that this perspective is also shared by computer scientists (Greff et al. 2020): “In the case of neural networks, the binding problem is not just a gap in understanding but rather characterizes a limitation of existing neural networks.”

Finally, We would remove ref11 from the references and include the work of Roelfsema as it is more consistent with our claim.

Our work on ViTs ties into a large and ongoing discussion about the capabilities of brains vs ANNs in object-based perception. We believe with the additional references we can make our relevance much clearer.

---

[W4.1, Q4] Weak Attention–Binding Correlation

We agree that the correlation in Figure 6 (r = 0.163/0.201) is modest, but it is consistently positive. However, such modest effects should be expected, after all attention does a lot more than binding. It is meaningful to plot attention weights against the estimated IsSameObject, as it reveals the model’s implicit belief about whether two patches belong to the same object.

To address the reviewer’s suggestion, we’ve extended our analysis to compare attention weights against the true patch‑pair labels, and find that attention is significantly higher for same‑object pairs than for any category of different‑object pairs.

Table 1: Mean self‑attention weights for different patch‑pair categories

[W4.2, Q4] Ambiguous ablation method and marginal effects

We’d like to clarify that “binding” part $b^l(x)$ is computed as $b^{(l)}(k) = h^{(l)}(x)^⊤W$, where W is the trained weight matrix of the quadratic probe. Our reasoning for this formulation is given in Lines 178–180. The ablation yields only modest performance gains (or losses), reflecting a relatively small effect size; we will explicitly note this as a limitation in the camera‑ready version’s limitations section.

---

[Q5] Baseline definition in Figure 3

The baseline corresponds to the accuracy of always predicting "different", which accounts for the class imbalance in the dataset.

---

[Q6] Extraction of binding vectors from token embeddings

We obtain the binding component $b_a$ by projecting $h_a$ onto the quadratic probe’s weight matrix $W$: $b_a = h_a^\top W,\quad f_a = h_a - b_a.$ Although this projection is an approximation, since the “true” binding vector was computed by subtracting embeddings of identical objects (Lines 231–237; Fig. 4), it performs well in practice, as the probe was trained on large, natural-image datasets rich in similar-looking objects that can only be distinguished by binding signals (Lines 178-180).

---

Minor concerns

We thank the reviewer for their careful review. We will correct the LaTeX/formatting issues and restructure the section as suggested to improve clarity and flow.

---

We believe that our responses have substantially strengthened the contributions of our work and are thankful for the reviewer's feedback.

Best regards,
Authors

Response to reviewer 43Cy(1/1)

Dear Reviewer 43Cy,

We thank the reviewer for recognizing our work's contribution in showing that training objective–based inductive biases can drive object binding independent of architectural design. We agree with their feedback and are delighted that we were able to run the suggested experiments (across different objectives and models) and that they substantially improved our paper.

Below, we address their comments point by point.

---

[W1, Q1] Limited verification across pretraining objectives

We have now extended our analysis beyond DINOv2 to include CLIP, MAE, and supervised-only trained ViTs. (Some limited experiments with CLIP and DINO model sizes were already included in the supplementary: Lines 471-478, Appendix Figure 7). To ensure fair comparisons, we now standardized the effective patch coverage by resizing inputs so that all models process the same spatial patch divisions. This ensures a shared baseline (72.6%) across all models (corresponding to the accuracy of always predicting "different," which accounts for the class imbalance in the dataset).

Here are our results:

Table: Probe accuracy on the IsSameObject task across ViTs

• Pretraining objectives: CLIP, MAE, and DINO all demonstrate object binding capabilities, supporting our broader claims about Large Pretrained Vision Transformers beyond DINOv2 alone. We also show that supervised ImageNet classification yields poor object‑binding performance, indicating that binding is an acquired ability under specific pretraining objectives rather than being a universal property of all vision models.
• Model size: Larger models maintain object binding effectiveness, but the optimal layer for object binding shifts earlier in the network (lower normalized layer index), with performance plateauing in later layers (see supplementary Lines 471-478, Appendix Figure 7). This suggests that increasing model scale makes object binding emerge earlier, and deeper layers become redundant for binding.

We offer explanations of why these training regimes work:

• DINO: As noted in Lines 186–189, the DINO loss drives the model toward object-level features by enforcing consistency between augmented views, since each view contains the same objects and the model must align their representations; this objective has been shown to encourage class and object specific features (Caron et al. ICCV 2021), as the reviewer points out.
• CLIP: By aligning images with text captions, CLIP effectively assigns each object a symbolic label (e.g., “the cat”), which can act like a pointer that pulls together all patches of that object. This supervision likely encourages patches from the same object to cluster in feature space. Future work could track object‑binding strength across CLIP training checkpoints to validate this hypothesis.
• MAE: The masked autoencoder objective requires the model to reconstruct a missing patch from its surroundings. When the masked patch sits between multiple objects, correctly predicting its content forces the model to infer which object it belongs to, thus promoting the grouping of patches from the same object.

Our findings thus produce a much wider coverage of ViTs and we provide an understanding of potential reasons for why binding emerges.

---

[W2] Object binding through intra-token interactions

As the reviewer correctly points out, multi‐head attention involves both intra‑token operations (the linear projections that mix channels within each token) and inter‑token operations (the dot‑product and weighted summation that relate different tokens). In our work, we define object binding strictly at the level of patch tokens, i.e. whether separate tokens group together to form a single object. Under this definition, no binding can occur within a single token alone, so intra‑token projections cannot contribute to the effect we study.

An alternative notion–binding different attributes within one token, such as “red” and “square”, would require us to decompose a patch’s embedding into separate feature channels. We appreciate that investigating this within‑token compositionality is an important direction, but it falls outside the scope of our current study because it is likely to use rather different mechanisms (for example, dealing with a relatively small set of tokens with a large number of intra-token feature channels).

---

[W3] Lack of evidence of compositionality

We appreciate the reviewer's comment on the importance of binding in functionally approaching human level generalization in neural networks. The reviewer references a seminal work (Greff et al. 2020), which defines binding as the ability to dynamically and flexibly bind information that is distributed throughout the network, they also investigate its functional aspects—segregation, representation, and composition—without implying that all of them are required by the definition itself. We agree these are useful perspectives, but note that the literature does not universally require compositionality as part of the core definition of binding.

The reviewer points out, while we clearly demonstrate (1) ViTs’ ability to group an object’s patches (i.e., segregation or parsing), we do not show (2) that those grouped features are disentangled into a truly compositional representation usable by downstream tasks. We appreciate this clearer framing and believe our results offer preliminary evidence of such compositional structure, yet we agree that a deeper exploration of compositionality lies ahead as future work.

To clarify,

• The distributed quadratic probe’s superior performance demonstrates that the IsSameObject signal spans many channels rather than being confined to a handful of specialized ones. While the reviewer interprets this distribution as evidence against compositionality, a distributed code can still be fully compositional and accessible to downstream tasks. Consider a signal that is intrinsically one‑dimensional but oriented along a rotated axis: when expressed in the original coordinate system, its nonzero components appear across multiple dimensions, yet its underlying structure remains compact.
• We distinguish between distributed representation and low intrinsic dimensionality of the IsSameObject space:
  • Distributed representation means the model uses several activation dimensions (channels) to encode whether two patches belong to the same object.
  • Low intrinsic dimensionality means that, despite this distribution, all those activations lie on a simple, low‑dimensional manifold or subspace—so the binding information itself is inherently compact. We demonstrate this low‑dimensional clustering in Lines 231–237.
• We acknowledge that a deeper investigation of compositionality in downstream tasks remains an important direction for future work. The empirical evidence for the compositionality of the feature representations is accumulating (Amir et al., Deep ViT Features as Dense Visual Descriptors, ECCVW 2022; Darcet et al., Vision Transformers Need Registers, 2023).

We believe that our responses have substantially strengthened the contributions of our work and are thankful for the reviewer's feedback.

Best regards,
Authors