Objective drives the consistency of representational similarity across datasets

We thank the reviewer for acknowledging that the paper is well-written and that our claims are supported by the results, which are presented clearly. We are grateful for the valuable feedback that helped us improve our paper. We will address each concern point by point. All new figures/tables (R4F1-F7, R4T1-T2) are available here.

First, we want to address [...] metrics might reflect the choice of datasets rather than similarities between the models. Our analysis included 23 datasets, mostly part of the VTAB [1] and commonly used in CV. We compute the correlation of similarity values for all pairwise combinations of datasets. Therefore, we obtain similarity consistency measures for all dataset combinations (e.g., natural vs natural, structured vs structured, natural vs structured, …). These are responsible for the variance shown in our boxplot (Fig. 5). According to the reviewer’s suggestion, we further analyzed the observed variance in Fig. 5 and isolated the effect of different dataset types. We created two new boxplots (Fig. 5), one containing only correlation coefficients where both datasets contain natural images (Fig. R4F1) and one containing only coefficients for specialized+structured datasets (Fig. R4F2). The main observation remains the same: irrespective of the dataset selection, the training objective is a major driver for representational similarity consistency. However, we observe small differences such as a smaller minimum correlation for natural images (see IN1k/XLarge & IN1k/Large) and a slightly larger variance for specialized and structure data (Fig. R4F2 vs Fig. R4F1). We will add the new plots to the camera-ready version’s appendix.

Second, we will address the relevance of sigma for CKA values computed with RBF kernel. We normalize all feature representations, leading to comparable distances over datasets. However, we agree that the strong correlation between CKA RBF 0.4 and CKA linear suggests that $\sigma=0.4$ (partly) captures global similarity structures. Fig. 9 shows small differences in the upper right corner; however, we agree that the difference between CKA linear and CKA RBF 0.2 is more pronounced. We extend our correlation plot (Fig. 3) by CKA RBF 0.2 as shown in Fig. R4F3. As expected, CKA RBF 0.2 differs more significantly from CKA linear. Based on this evidence, we agree with your perspective that RBF 0.4 still captures some global similarity while RBF 0.2 is a better candidate for analyzing local similarity. Thus, we repeated our experiments with the local kernel (see Fig. R4F4 and R4F5). We observe the same overall pattern, indicating that the objective is relevant for similarity consistency, while network architecture is less important. This analysis reveals two interesting observations. For local similarity, the supervised models are more consistent than the Img-Txt models, which are better at recovering global structure. In addition, models trained on IN-21k are more consistent than models trained on IN-1k, of which the latter set is more consistent in their global structure. IN21k contains substantially more classes (21.843) and a higher percentage of the classes are leaf nodes in the WordNet tree (76.71% vs 65% for IN1k), representing more fine-grained entities. The representation must contain more fine-grained details to distinguish these classes, dominating local similarities. We will include these observations in Appx. F of the camera-ready version.

Third, we investigate the stability of CKA values across different subsets. Fig. 10 shows that CKA linear is stable with 5k samples, showing smaller variance when increasing to 10k samples. To further analyze this stability, we performed bootstrapping over 500 subsets of IN-1k features, each containing 10k samples (as suggested), for 6 example models (the anchor models from Appx. G). Fig. R4F6 and the narrow confidence intervals (CI) in Tab. R4T1 confirm the stability of CKA linear values, demonstrating minimal variation across bootstrapped subsets. For RBF 0.2, Fig. 10 showed instability with a subset of size 10k, while 30k samples are more robust. We verified this with another bootstrapping experiment, Fig. R4F7, and the CIs in Tab. R4T2 showed a larger variance in CKA values. This suggests that local similarity depends more on the specific stimuli than global similarity. This is not surprising as stimulus-specific fine-grained details drive local similarity measurements. To mitigate this effect, we increased the number of samples for experiments analyzing local similarity from 10k to 30k when the dataset size permitted.

Last, we agree that the t-SNE Figure helps us understand our categorization and will move this visualization from the appendix to the main part.

[1] Zhai, Xiaohua, et al. "A large-scale study of representation learning with the visual task adaptation benchmark." arXiv preprint arXiv:1910.04867 (2019).

We would like to thank the reviewer for the valuable feedback and appreciate the assessment of our manuscript as being well-written and interesting work.

First, we agree that the individual components of our analysis framework are well-established rather than novel by themselves. We see this as a strength rather than a weakness because it allowed us to build a novel meta-framework relying on the methodological soundness of the individual components without introducing subcomponents that still need to be validated by the research community. To the best of our knowledge, our work is the first to propose a structured way of using these components to characterize how similarity relationships vary across datasets. Our main focus lies on introducing and formalizing similarity consistency and conducting a large-scale analysis of this measure across 64 different vision models and 23 datasets. We identify that the learning objective is a crucial driver of similarity consistency, and architectural specifications before training (architecture type and size) are less relevant.

Second, while confounding factors cannot be avoided when evaluating the limited set of large, well-trained general vision models, we tried to unravel some confounders in Appx. G and H.

• In Appx. G, we selected two anchor architectures (ResNet-50 and ViT-L) and systematically varied the training objective (Sup., SSL, Img-Txt) while keeping the architecture type and model size constant, resulting in six models. This allowed us to isolate the effect of the objective in a more limited but controlled setting. We observed that even here, the training objective appears to be driving similarity consistency.
• In Appx. H, we investigate the role of training data on similarity consistency. To that end, we fixed the objective (supervised) and network architectures (AlexNet, DenseNet161, ResNet18, and ResNet50) and varied the training dataset (general-purpose vs. domain-specific). Here, we identified higher consistency in models trained on a domain-specific dataset (Places365), most likely due to a more constrained solution space in comparison to general-purpose models trained on large-scale datasets. These findings complement our main analysis, which deliberately focused on general-purpose vision models trained on datasets with large semantic diversity.

Last, we fixed the Fig. reference in Section 4.3. We intended to refer to Fig. 11 in the Appx. D. We will correct this in the camera-ready version by using the extra page to move this figure back to the main text.

First, we thank the reviewer for their overall positive feedback and for pointing us to two papers that helped strengthen the integration of our work into the existing literature. We agree with their relevance to our work due to using Representational Similarity Analysis [RSA; 2] or other similarity measures [1] to (pre-)select (downstream) task-specific models. Therefore, we will cite them in the related work section of our manuscript’s camera-ready version: “[...] However, recent work demonstrated that representational similarities (e.g., measured via RSA) can be used to effectively select (downstream) task-specific models [1,2].”

Second, we agree on the lack of clarity of some of the figure captions and will use the extra page in the camera-ready version to expand these captions to improve clarity, i.e., for Fig. 5.

In addition, we thank the reviewer for posing three interesting questions, which we will elaborate on, providing additional results (R3F1-F5) in an anonymized repository.

Q1: Which SSL models are close to Img-Text in Fig. 4, and why?

Fig. R3F1 zooms into the area of Img-Txt models for the three dataset combinations of Fig. 4 and labels individual model pairs. We observe that many SSL and image-text model pairs show high CKA values across the datasets, i.e., are located in the upper right corner and therefore in proximity. These models are similar within each objective. The SSL model pairs show high similarities as they are trained with similar datasets, augmentations, and losses (e.g., BarlowTwins/VicReg and MoCov2/SimCLR). Some image-text models are quite similar as well. However, the proximity of these two model pair sets does not allow us to infer any direct relationship between them. For this, we must consider the similarities of model pairs containing both model types, as shown in Fig. R3F2. The points, representing cross-type similarities between SSL and Img-Txt models (pink), have lower correlations than within-type pairs. This indicates that despite some individual SSL models appearing close to Img-Txt models, the overall relationship between these two categories of models is less strong and more variable.

Q2: Could Fig. 2 and 4 be provided for more specialized datasets?

We thank the reviewer for their suggestion and have decided to include additional figures extending Fig.2 with datasets of each category (Fig. R3F4) and Fig.4 containing the EuroSAT and DTD (Fig. R3F5).

Q3.1: Do you attribute the inconsistency patterns in CIFAR-10 and CIFAR-100 primarily to their significantly lower image resolution compared to IN-1k?

Yes, we indeed think that CIFAR-10/100’s small image size (32×32 pixels) restricts the representation space by severely limiting fine-grained details and surrounding contextual information. Appx. I shows clear differences in correlation coefficients between performance gaps (def. Sec. 4.7) and CKA values: low absolute correlation for IN-1k versus high for CIFAR. This suggests IN-1k’s high-resolution images (224×224 pixels) provide rich contextual cues that support diverse, well-performing representations, whereas CIFAR’s constrained resolution and minimal background details restrict the range of viable representations.

Q3.2: Can you elaborate on the underlying factors causing the visible clustering patterns observed for certain datasets in Fig. 6?

Looking more closely at the clustering patterns of Fig. 6, we observe:

• CIFAR-10 and CIFAR-100 form a strong cluster, potentially due to their low-resolution format and similar categorical structures, showing also high consistency in the (Img-Txt, Sup).
• The Breeds datasets, Caltech-101, Country-211, STL-10, and Pets cluster together based on their similar domain properties, resolution profiles, and centered object compositions, showing also high consistency in the (Img-Txt, Sup).
• Medical imaging datasets do not cluster together, potentially due to their fundamentally different visual patterns and domain-specific features (eye vs. tissue scans).
• RESISC45 shows stronger correlations with structured datasets than with other specialized datasets (i.e., (Img-Txt, Sup)). This cross-category relationship might stem from satellite imagery's inherent structural properties—regular grid layouts, transportation networks, and geometric patterns—which create feature distributions resembling those in structured datasets. However, it differs from EuroSAT potentially due to RESISC45's higher resolution imagery, greater geographic diversity, and more diverse class set.

A further analysis that isolates natural images from structured and specialized datasets can be found in the first part of Reviewer 9H2r's answer.

We will incorporate the main points of these answers into our camera-ready version.

We would like to thank the reviewer for their helpful suggestions and for acknowledging the relevance of our work and the soundness of our experiments. We believe that following the reviewer’s suggestions allowed us to notably improve our analyses. Two points stood out in particular, which we address in detail below. We added all additional figures (R1F1-F5) to an anonymized repository and refer to the specific Figures detailed in the README.md.

First, we would like to clarify the range of the covered training objectives. In representation learning for computer vision, self-supervision (SSL) and image-text alignment are currently the state-of-the-art approaches. Therefore, we selected these two categories alongside supervised learning. The SSL group contains a diverse set of objectives, including self-supervised contrastive losses (such as SimCLR), the mentioned masked image modeling (MAE), self-distillation (DINO), extended self-distillation (DINOv2), but also pretext-task-based (Jigsaw, RotNet), redundancy-reduction (BarlowTwins) and clustering-based (SwAV) losses*. The total set of 64 models contains (most) SOTA models for representation learning in vision. For this diverse set, we identified the training objective as a crucial factor for the consistency of representational similarity, while model architecture and size appear less relevant.

As we identified the training objective as a main driving factor for similarity consistency, we are convinced that due to its flexibility, our framework can, in future work, easily be applied to analyze other training objectives, e.g., as recommended, testing the effect of robustness losses on the similarity structure of out-of-distribution data.

Second, we would like to address the effect of taking the final layer of the model by evaluating the representational similarity consistency on intermediate layers. In our work, we followed the standard procedure of extracting features of the final layer (or penultimate layer, for classification models), which is commonly referred to as the representation [1]. We remark that these representations are of special interest because they are the ones used for downstream tasks. However, we agree with the reviewer on the potential role of layer choice for our findings, as mentioned in our discussion section. Our proposed analysis framework does not depend on specific layers; it can also be applied to intermediate layers. Therefore, we followed the reviewer's suggestion and repeated the consistency analysis for the middle layers of a large subset of our transformer models. We omitted CNN-based models, as middle-layer extraction is less clear when representations depend on spatial location. We remain consistent in our extraction method across layers for the transformer models: If the model's original representation was derived from the classifier token, we also extracted the classifier token from intermediate layers. Otherwise, we applied avg. pooling. Fig. R1F1 and R1F2 show similarity matrices, analogous to Fig. 2. Here, model representations tend to be more similar across models of the same type. W.r.t. the consistency of similarities, we observe a lower median and larger standard deviation of consistency of representational similarities across all model pairs (grey bar) in Fig. R1F5 compared to Fig. 5. While the variances are relatively large for the training data and model size categories, we see above-median consistency for within-objective model pairs. We speculate that higher consistencies within training objectives in intermediate layers indicate that the training objective has a stronger influence on representational structures already early in the network. For example, supervised models may form structurally similar lower-level representations but are less constrained in the organization of their representations close to the classification output.

In conclusion, our analyses of intermediate layers support the finding that the training objective is a main influence on representational similarity consistency, though more experiments would be needed to fully characterize layer-specific effects. We thank the reviewer for their insightful comments that have strengthened our analysis.

* References to the models can be found in the main paper, in Tab. 2.

[1] Kornblith, Simon, et al. "Do Better ImageNet Models Transfer Better?" CVPR, 2019.