SynBench: Evaluating Pretrained Representations for Image Classification using Synthetic Data

7. How can SynBench predict classification performance across a broad range of tasks

Think of how representation learning research typically evaluate a model for transfer learning - by running tests on broad range of downstream tasks. And the reason behind this is to see how the model behaves in different scenerios. To theorize things, we believe the general behavior of a pretrained representation is measured by how it perform on tasks of different difficulty levels. That is why we think a fundamental part of our design is to simulate tasks of different difficulty levels. One difference between SynBench and a traditional probing test is that, for example, we are using the classification problem of two highly overlapped Gassuain, instead of classifying ImageNet21k. We hope this clarification builds enough intuition to understand the following:

1. We vary $s$ in equation 2 from 0.1 to 5 in increments of 0.1, which correspond to optimal accuracy (groundtruth difficulty) ranging from 55% to 100% and 50 difficulty levels. If we refer to this figure, we see each of the red points correspond to one of our similated trials with difficulty levels (x-axis).
2. Baseline methods are task/data dependant, which means they are somewhat bound to tasks of that similar difficulty levels. If we refer to the same figure, it could be the single purple point with fixed level of difficulty.
3. If we include certain knowledge of possible downstream data properties, say locality of pixel dependencies, then the prediction will indeed be more accurate (see our section 4.4).

4. SimCLRv2 has higher SynBench-Score than DINO

We need to note that SynBench is to evaluate specific models, not methodology. As there might have many subtle differences in training parameters/optimizers/datasets, we can only make conclusions for models used in our paper. In our paper, we use SimCLRv2 models from the simclr official repository, specifically, we use SimCLRv2 models with resnet depth 50 and 100, both with width 2X and SK True. For example, if we try to compare with DINO models, we can refer to DINO official repository and see they are generally on par with the specific SimCLRv2 models we considered (see table 4.2). From this example, we don't see these two specific SimCLRv2 models to perform more poorly than DINO, and our paper makes no conclusions for BYOL and MAE. As such, we do not see there is a problem for our proposed method.

Table 4.2 The linear probing accuracy on ImageNet1k reported in the original repository.

5. CIFAR10 and TinyImageNet as "real-life downstream data" in Section 4.3

These are popular CV downstream datasets. We have updated our paper accordingly.

6. GMMs' capability of modeling the statistics of natural images

We believe the claim the reviewer is referring to is "Our use of Gaussian mixtures for analysis is supported by its capability of modeling the statistics of natural images (Zoran & Weiss, 2012) and prior arts ......". Since our target is pretrained vision models, the capabilities of preceiving contrasts and edges etc are centric (this is also the motivation for convolutional kernels for ResNets, images patches for ViTs etc). From the reference, we quote "Simple Gaussian Mixture Models (GMMs) learned from pixels of natural image patches have been recently shown to be surprisingly strong performers in modeling the statistics of natural images" and it is not hard to see that these statistics include contrast, textures at different scales and orientations, and boundaries of objects in the reference. If the reviewer is curious about how images patches connects to whole images, there are some discussions in the literature, From Learning Models of Natural Image Patches to Whole Image Restoration and From Patches to Natural Images via Hierarchical Dirichlet Processes. We note that another reason for these seminal works to study from image pathes instead is "Due to the high dimensionality of the images captured by modern cameras" (quote from From Patches to Images: A Nonparametric Generative Model). Afterall, at the bare minimum we all know GMMs are universal approximators of densities (Universal Approximation Using Radial-Basis-Function Networks, Networks for approximation and learning), meaning any smooth density can be approximated with any specific nonzero amount of error by a Gaussian mixture model with enough components.

To our knowledge, we don't know which part is misleading but we are willing to remove or modify the sentence ("Our use of Gaussian mixtures for analysis is supported by its capability of modeling the statistics of natural images (Zoran & Weiss, 2012) and prior arts ......") following the reviewer's suggestion.

1. Why Gaussuian distribution

We view SynBench as a "necessary" and "minimum" model test. We concur with the reviewer's argument that assuming a Gaussian distribution for raw input data in pixel space implies starting with a significantly structured and "good" representation. This assumption is intentional and serves the specific purpose of establishing a foundational test. If a pretrained representation network cannot preserve the quality of data drawn from such simple, well-defined distribution (indicated by a SynBench-score close to 1), it raises significant concerns about its potential performance in more complex, real-world scenarios. In our test, we do see that the seperability of the representations could degrade by significant margins compared with their reference behavior in the well-defined simple input space.

To draw an analogy for easier understanding, consider a hgh-performing language model. One might test its basic capabilities by checking whether it correctly writes "2" after "1+1=" or classifies "bad" as a negative word. Failure to perform these straightforward tasks with clear expected outcomes would suggest fundamental deficiencies in the model.

Moreover, we also believe that our designed task with conditional Gaussian is feasible. For a feature extractor robust to noises/good at denoising, all class-1 synthetic samples shall be treated as point $\mu$ and all class-2 samples as point $-\mu$, which are then trivial to classify.

2. Pearson and confidence interval

Let $r$ be the Pearson correlation coefficient, $p$ be the number of models. We ran the calculation based on the reference shared by the reviewer, and see that the upper and lower confidence interval limits in z-space are $0.5\ln(\frac{1+r}{1-r}) \pm 1.96\sqrt{\frac{1}{p-3}}=1.589\pm 1.386$. Translating to r-space by $r=\frac{e^{2z}-1}{e^{2z}+1}$ yields the upper limit of 0.995 and the lower limit of 0.203. That said, we believe the confidence interval for the Pearson correlation of 0.92 is instead $[0.203,0.995]$ if the desired confidence level is $95\%$. We are not sure how the reviewer obtained $[-0.81,1.0]$ and wish the reviewer can let us know to make sure we are on the same page.

In the following Table 4.1, we added four efficient nets' SynBench-scores, together with the average of their reported performance on 27 downstream tasks in Radford et al., 2021, Table 10. We ran the same calculation for the Pearson correlation coefficient $r=0.87$ and $p=9$ to obtain the confidence interval of $[0.488,0.972]$ which suggest at least moderate correlation upto strong corelation.

Table 4.1 The correlation between SynBench-score and the average accuracy on 27 real-life tasks.

3. Role of hyperparameters

Threshold accuracy $a_t$. $a_t$ is a user-specified threshold accuracy based on their utility requirement. To give a complete picture, we calculate all SynBench-score from 0.7 to 0.9 in table 6 in the paper. By referring to the table, users can see which model might be more suitable for him/her. Throughout our analysis, we used $a_t=0.7$ since in average the reallife tasks accuracy is above 70% (Table 7 in the paper).

Robust margin $\epsilon$. Recall that $\epsilon$-robust Bayes optimal classifier give the optimal classifier if the expected perturbation norm is subject to $\epsilon$ budget. When SynBench is used to evaluate clean accuracy without adversarial attacks as in Section 4.2, we let $\epsilon=0$. When we want to perform $\epsilon$-robust linear probing to increase the robustness against adversarial examples as in Section 4.3, we let $\epsilon=\text{argmax}_\epsilon \text{SynBench-Score}$.

Attack strength. Throughout our analysis, we let the attack strength be 0.2. This is guided by our wish to distinguish each model's robustness level to adversarial attacks. If the attack strength is too strong, the robust accuracy of all models will decrease to near zero, while if the attack is too weak, no misclassification will occur. We note that this hyperparameter is only for attack evaluation, and it won't affect the computation of SynBench score.

Dear Reviewer af4U,

We would like to thank you again for your review! As the rebuttal/discussion phase is nearing its end, we want to check in to see if we have dispelled all your concerns, or if there are any additional points we can help clarify.

We understand that the discussion period is short, and your support and feedback are very important to us. We are truly grateful for your insights and the time you've dedicated to reviewing our work.

Thank you for your detailed response! Apologies for getting back to you so late.

My fundamental issue remains. The paper is making a rather extraordinary claim: that assessing performance on separating two Gaussians could be predictive of performance on arbitrary downstream classification tasks. I don't see enough strong evidence to support that extraordinary claim. Some specific responses can be found below.

Regarding the Gaussian distributions, I tend to agree with other reviewers that it seems like a fairly contrived task, and it is difficult to see why it would be predictive of performance on arbitrary downstream tasks. It seems like the authors more or less agree (with the "contrived" part, anyway) - based on the authors' response, SynBench is supposed to be a low bar that any reasonable representation should clear. But then shouldn't any model that performs decently well on real-world tasks excel on SynBench? If so, what are we to make of e.g. Table 4.1 in the rebuttal, where the EfficientNet backbones have awful SynBench scores but good real-world accuracies?

Regarding the confidence intervals, you are correct that I messed up the calculation and I agree with your numbers. However, I think my point stands. A key claim of the paper - "The Pearson correlation coefficient between SynBench-Scores and the average real-life task accuracy is larger than 0.9" - is pretty misleading if the $95\%$ confidence interval is $[0.203, 0.995]$. Moreover, with such a wide confidence interval, this does not seem to be strong evidence that the proposed approach works much better than other approaches. Based on the confidence intervals we're discussing, it seems like $n=5$ is likely too small to draw strong conclusions about Pearson's $r$. While the additional results in Table 4.1 are interesting, there is no comparison to other methods. The scores also vary wildly between backbones that have similar real-task performance, which raises more questions that it answers.

Regarding GMMs and natural images, you are correct that I'm referring to the claim that the paper's claim that the "use of Gaussian mixtures for analysis is supported by its capability of modeling the statistics of natural images". Since this paper uses Gaussian mixtures very differently than Zoran & Weiss (i.e. two Gaussians for the whole image vs. tens of Gaussians for modeling patches) this strikes me as a misleading use of that reference. What evidence in Zoran & Weiss supports modeling entire natural images using a mixture of a small number of Gaussians?

Regarding model ranking, I didn't realize the SimCLR models were 2x variants. Probably it's best to update the tables to rename e.g. "Resnet50-SimCLRv2" to something like "Resnet50(2x)-SimCLRv2", since "Resnet50" by itself typically just denotes the 1x version. But there are still ranking reversals - why should the ResNet101 variant have a higher SynBench score than the ResNet50 variant, when the 101 has better performance than the 50? It seems like an important point to understand.

At the moment I lean towards keeping my rating as-is - I think the paper has potential, but currently there are too many question marks that have direct consequences for the central claims of the paper.

We thank the reviewer for getting back to us. We are also glad that we still have a little time to run the additional results the reviewer wants to see and share more of our thoughts! Your comments and discussions are intrumental to our work.

Before we provide our detailed responses, we would like to use this opportunity to reiterate the overarching goal and contributions of this paper. Our primary objective is to introduce a new task-agnostic framework for evaluating the robustness-accuracy of pretrained representations and to inspire other task-agnostic benchmarking designs. This paper already encompasses a substantial amount of material (what is the metric, how can we use the metric to inform performance, how can we use the metric to select robust linear probing parameters), and we believe it is prudent to reserve some aspects for future exploration and development, including improving its rank efficiency (though SynBench already performs the best among all compared baselines).

what are we to make of e.g. Table 4.1 in the rebuttal, especially EfficientNet backbones

We want to draw to the reviewer's attention that EfficientNet backbones perform almost the worst on average (except ViT-B/32), and also on some tasks (e.g. Food101, CIFAR10, CIFAR100, SUN397, Caltech101, Flowers, FER2013, Country211, UCF101, Kinetics700). If we check their SynBench scores, we see, when $n=8192$, SynBench scores of EfficientNets are around 0.1 while ViT-B/32 is 0.01. By that, we think SynBench indeed informs the general performance.

baselinse with additional models

Thank you for confirming with us and we are glad that we are on the same page regarding the calculation. In the following table, we have added the baseline results to make Table 4.1 more complete and have also included the equivalent figure of Figure 4 of the paper here.

Table 4.1R The correlation between SynBench-score and the average accuracy on 27 real-life tasks.

GMMs and natural images

We propose to revise the sentence as

In an earlier work that discusses the connections between GMMs and natural image modeling, it has been demonstrated that "Simple Gaussian Mixture Models (GMMs) learned from pixels of natural image patches have been recently shown to be surprisingly strong performers in modeling the statistics of natural images". Our adoption of Gaussian mixtures for analysis is inspired by this finding, as the image patches in our synthetic data are also Gaussians. Additionally, the use of Gaussian is further supported by their capability to act as universal approximators of densities as well as by existing literature on Gaussian design.

We are open to modifying or removing this sentence should the reviewer find it necessary.

1. How SynBench would perform in the finetuning setup

As discussed in specified in Section 3.3, SynBench is designed to evaluate the pretrained representations parameterized by $\theta$. In the scenerio when finetuning is performed, the parameters of the representation network will be updated, hence resulting in another pretrained representations (a separate model) from the perspective of SynBench. In our paper, we have shown how to use SynBench to compare representation networks before and after finetuning (Section 4.2, Comparing model attributes). In our example, ViT-B/16 and ViT-B/16-in21k were both pretrained on ImageNet21k with supervision, whereas ViT-B/16 is further finetuned on Imagenet 1k (name convention is adopted from PyTorch Image Models). The result shows the utility of SynBench in comparing original and finetuned models.

2. The number of test samples in the synthetic dataset

There are 2048 test samples in the synthetic dataset. We have included this detail to the updated paper.

3. $z_1$ and $z_2$ in Line 12 of Algorithm A.6

The subscript denotes the class label of the representation. That is, $z_1^{train,i}$ is the $i$-th training sample from class 1, and $z_2^{train,j}$ is the $j$-th training sample from class 2. We have included this detail to the updated paper.

4. How does SynBench inform model performance on OOD tasks vs. others

We thank the reviewer for raising this question. In the following Table 2.1, we computed the Frechet Inception Distance (FID) scores from ImageNet21k to the downstream tasks, and used them as the indicator of how OOD are the tasks. We then computed SynBench-score correlation with tasks that have FID scores larger than a threshold {50,100,150,200}. We do want to note that not all models in our analysis are pretrained with ImageNet21k; however, since ImageNet21k has become a go-to pretraining dataset, we assume samples therein are in-distribution.

From table 2.2, we see that if we don't apply filter on FID (or equivelantly let threshold be 0), the initial correlation was 0.92. As we gradually increase the threshold to 50, 100, 150, and even 200, the correlation stays above 0.8, indeed suggesting SynBench's robustness to OOD tasks.

Table 2.1 The Frechet Inception Distance (FID) scores from ImageNet21k to 27 downstream tasks.

Table 2.2 The correlation between SynBench-score and the average accuracy of FID-thresholded downstream tasks.

Dear Reviewer 97TE,

We would like to thank you again for your review! As the rebuttal/discussion phase is nearing its end, we want to check in to see if we have dispelled all your concerns, or if there are any additional points we can help clarify.

We understand that the discussion period is short, and your support and feedback are very important to us. We are truly grateful for your insights and the time you've dedicated to reviewing our work.

1. How does SynBench inform model performance on OOD tasks vs. others

We agree that showing SynBench's robustness to OOD tasks will strengthen the applicability and reliability. In the following Table 1.1, we computed the Frechet Inception Distance (FID) scores from ImageNet21k to the downstream tasks, and used them as the indicator of how OOD are the tasks. We then computed SynBench-score correlation with tasks that have FID scores larger than a threshold {50,100,150,200}. We do want to note that not all models in our analysis are pretrained with ImageNet21k; however, since ImageNet21k has become a go-to pretraining dataset, we assume samples therein are in-distribution.

From table 1.2, we see that if we don't apply filter on FID (or equivelantly let threshold be 0), the initial correlation was 0.92. As we gradually increase the threshold to 50, 100, 150, and even 200, the correlation stays above 0.8, indeed suggesting SynBench's robustness to OOD tasks.

Table 1.1 The Frechet Inception Distance (FID) scores from ImageNet21k to 27 downstream tasks.

Table 1.2 The correlation between SynBench-score and the average accuracy of FID-thresholded downstream tasks.

2. Whether SynBench maintains its effectiveness when applied to larger and more complex datasets

When we check how SynBench can serve as a performance metric of pretrained models, we used the average accuracy of 27 downstream tasks as the proxy of the general performance. Among the 27 tasks, there are indeed datasets that are large and complex, inclduing ImageNet. In the following Table 1.3, we highlight 3 subsets of tasks that represent more challenging datsets in different dimensions (number of classes, data types, task types).

1. For datasets that have more than 100 classes (Food101, Birdsnap, SUN397, StanfordCars, Aircraft, Caltech101, Flowers, Country211, UCF101, Kinetics700, ImageNet), SynBench-score correlates with their average performance with correlation of 0.56, compared with the best baseline (SFDA) of 0.19.
2. For video datasets (UCF101 and Kinetics 700), SynBench-score correlates with their acerage performance with correlation of 0.72, compared with the best baseline (SFDA) of 0.36.
3. For the visual reasoning and question-answering dataset, CLEVR,, SynBench-score correlates with its performance with correlation of 0.72, while LogME and SFDA demonstrate even stronger correlation ($>0.9$).

Overall, SynBench shows robust performance across these break-down groups.

Table 1.3 The correlation between SynBench-score and subsets of downstream tasks.

Dear Reviewer S4rE,

We would like to thank you again for your review! As the rebuttal/discussion phase is nearing its end, we want to check in to see if we have dispelled all your concerns, or if there are any additional points we can help clarify.

We understand that the discussion period is short, and your support and feedback are very important to us. We are truly grateful for your insights and the time you've dedicated to reviewing our work.

Subset of more challenging tasks

We further analyze SynBench score's correlation to the subset of more challenging tasks. When we check how SynBench can serve as a performance metric of pretrained models, we used the average accuracy of 27 downstream tasks as the proxy of the general performance. Among the 27 tasks, there are indeed datasets that are large and complex, inclduing ImageNet. In the following Table 3.4, we highlight 3 subsets of tasks that represent more challenging datsets in different dimensions (number of classes, data types, task types).

1. For datasets that have more than 100 classes (Food101, Birdsnap, SUN397, StanfordCars, Aircraft, Caltech101, Flowers, Country211, UCF101, Kinetics700, ImageNet), SynBench-score correlates with their average performance with correlation of 0.56, compared with the best baseline (SFDA) of 0.19.
2. For video datasets (UCF101 and Kinetics 700), SynBench-score correlates with their acerage performance with correlation of 0.72, compared with the best baseline (SFDA) of 0.36.
3. For the visual reasoning and question-answering dataset, CLEVR,, SynBench-score correlates with its performance with correlation of 0.72, while LogME and SFDA demonstrate even stronger correlation ($>0.9$).

Overall, SynBench shows robust performance across these break-down groups.

Table 3.4 The correlation between SynBench-score and subsets of downstream tasks.

4. Correlation breakdowns

As SynthBench score is not dependant on task, we gave the SynthBench score of each model in Table 7 in the paper. We calculate how SynthBench score correlates with downstream performance per data set in the following Table 3.1.

Table 3.1 The correlation between SynBench-score and individual downstream task.

Subset of OOD tasks

We further analyze SynBench score's correlation to the subset of OOD tasks. In the following Table 3.2, we computed the Frechet Inception Distance (FID) scores from ImageNet21k to the downstream tasks, and used them as the indicator of how OOD are the tasks. We then computed SynBench-score correlation with tasks that have FID scores larger than a threshold {50,100,150,200}. We do want to note that not all models in our analysis are pretrained with ImageNet21k; however, since ImageNet21k has become a go-to pretraining dataset, we assume samples therein are in-distribution.

From table 3.3, we see that if we don't apply filter on FID (or equivelantly let threshold be 0), the initial correlation was 0.92. As we gradually increase the threshold to 50, 100, 150, and even 200, the correlation stays above 0.8, indeed suggesting SynBench's robustness to OOD tasks.

Table 3.2 The Frechet Inception Distance (FID) scores from ImageNet21k to 27 downstream tasks.

Table 3.3 The correlation between SynBench-score and the average accuracy of FID-thresholded downstream tasks.

1. SynBench's target

SynBench's target covers both adversarial robustness and representation seperability (accuracy). To be more precise, we believe a good pretrained representation network should produce representations that entail good seperability, either with or without, adversarial perturbations. Please see our paper section 1, the second paragraph, for motivations of our scope. SynBench-score as a metric is designed to serve this purpose and perform evaluations. By definition, SynBench-score is the relative AUC of expected bound-threshold accuracy curve (see equation 3). When SynBench is used to evaluate clean accuracy as in Section 4.2, we let $\epsilon=0$, whearas when it is used to suggest robust linear probing parameters $\epsilon$ as in Section 4.3, we let $\epsilon=\text{argmax}_\epsilon \text{SynBench-Score}$.

2. How can SynBench predict classification performance across a broad range of tasks

Think of how representation learning research typically evaluate a model for transfer learning - by running tests on broad range of downstream tasks. And the reason behind this is to see how the model behaves in different scenerios. To theorize things, we believe the general behavior of a pretrained representation is measured by how it perform on tasks of different difficulty levels. That is why we think a fundamental part of our design is to simulate tasks of different difficulty levels. One difference between SynBench and a traditional probing test is that, for example, we are using the classification problem of two highly overlapped Gassuain, instead of classifying ImageNet21k. We hope this clarification builds enough intuition to understand the following:

1. We vary $s$ in equation 2 from 0.1 to 5 in increments of 0.1, which correspond to optimal accuracy (groundtruth difficulty) ranging from 55% to 100% and 50 difficulty levels. If we refer to this figure, we see each of the red points correspond to one of our similated trials with difficulty levels (x-axis).
2. Baseline methods are task/data dependant, which means they are somewhat bound to tasks of that similar difficulty levels. If we refer to the same figure, it could be the single purple point with fixed level of difficulty.
3. If we include certain knowledge of possible downstream data properties, say locality of pixel dependencies, then the prediction will indeed be more accurate (see our section 4.4).

3. Discussions: dependency on the properties of the pretrained feature extractor

We thank the reviewer for bringing this up. If possible, can the reviewer please share the literatures mentioned that use properties of the pretrained representations in similar contexts with us? In our theory, the reference optimal behavior is completely independant of the pretrained feature extractor (the denominator in Equation 3), and the evaluation depends on the feature extractor via the numerator. During this process, we do not explicitly use properties such as Lipschitzness. To our knowledge, the connection between Lipschitzness of the representation network and the downstream task accuracy via linear probing is not obvious.

I thank the authors for their effort in providing a detailed rebuttal. Overall the provided rebuttal material confirmed my original opinion. Here are responses to some of the author's points in the rebuttal.

2. How can SynBench predict classification performance across a broad range of tasks

I stand by my original assessment that some information about the downstream tasks is required to make reasonable predictions about the performance of predictors on these tasks. A mixture of two Gaussians in pixel space is an extremely inaccurate model of natural images. As an insightful example the authors can compute the SynBench for the identity function (i.e. directly in pixel space, without any feature extractor) and compare its performance with features extracted with strong image classification models.

3. Dependency on the properties of the pretrained feature extractor

The very first paper on adversarial examples (Szegedy et al., 2013) analyzes the stability of the network as a function of the Lipschitz constants of the individual network layers. Many follow-up works rely on similar properties (searching for "Lipschitz" in the 14.7k citing articles of (Szegedy et al., 2013) leads to 1.65k results on Google Scholar).

4. Correlation breakdowns

I thank the authors for providing the detailed breakdown. Unfortunately, this confirms that SynBench is not very predictive for the downstream performance across a broad range of tasks. The correlations for the individual data sets are very inconsistent: some are negative, some are positive, some are close to zero.

Dear Reviewer 4o5T,

We would like to thank you again for your review! As the rebuttal/discussion phase is nearing its end, we want to check in to see if we have dispelled all your concerns, or if there are any additional points we can help clarify.

We understand that the discussion period is short, and your support and feedback are very important to us. We are truly grateful for your insights and the time you've dedicated to reviewing our work.