Energy-based Automated Model Evaluation

We appreciate your positive comments and valuable suggestions.

W1: Concise description of temperature and small range temperature ablation

Thanks for your advice! The parameter $T$ enables control over the smoothness of the model's probability distribution during generation.

The ablation results for a wide range of temperature hyperparameter ($T$ from 0.1 to 100) are shown below:

In fact, despite the ideal case for MDE being T→0 in Theorem 3.1, our empirical results found that the performance variation caused by T from 0 to 1 is minimal. Thus, we fix T = 1 to ensure that the classification accuracy is not affected so that we can compare fairly with baselines. Most importantly, we did not rely on T to tune performance.

W2: Lacking a comparison in terms of evaluation time and memory usage between MDE and the training-must method

Honestly speaking, it is inaccessible to fairly compare the running time and memory usage of different methods.

Because the workflow of different methods are very different, e.g. training model from scratch, fine-tuning model, calculating model feature, statistic dataset distribution, computing the disagreement of ensemble prediction, etc.

Intuitively, it is easy to understand because the training-must method requires retraining the model, which can consume huge memory and require much more time, while MDE only requires computing values from the model logits.

So, we simplify this problem to compare the time complexity and space complexity of different methods in a rough granularity:

1. For time complexity:

   AC = ANE < ATC < AvgEnergy = MDE(ours) = NuclearNorm < Frechet < ProjNorm(training-must) < AgreeScore(training-must)

2. For space complexity:

   training-free(AC, ANE, ATC, AvgEnergy, MDE(ours), NuclearNorm, Frechet) < training-must(ProjNorm, AgreeScore)

We hope our response addresses your original questions. Please let us know if you have any other comments. We are eager to hear your feedback.

We express our gratitude for favorable feedback and valuable insights, and we sincerely hope that our response can address your concern.

W1: No source code provided.

We apologize for the misunderstanding caused by not mentioning in the paper that we have provided our source code in the supplementary materials.

W2: Interpretability can be further explored.

Thanks for your comments. We will engage in further discussions to enhance the interpretability of our methods.

For instance, we will introduce more easily comprehensible theoretical proofs, provide concrete examples/illustrations, and showcase a greater number of visual experiments in our future work.

W3: The comparison of time costs among different methods is essential.

Honestly speaking, it is impossible to fairly compare wall clocks of different methods.

Because the workflow of different methods is very different, e.g. training the model from scratch, fine-tuning the model, calculating model features, statistic dataset distribution, computing the disagreement of ensemble prediction, etc.

So we simplify this problem to compare the time complexity of different methods in a rough granularity:

1. From the view of method categorization:

   confidence-based < regression-based (we are here) < based on data distribution shift = based on model feature < based on model parameter shift < disagreement-based = based on self-supervised task

2. From the view of the compared baseline in this article:

   AC = ANE < ATC < AvgEnergy = MDE (ours) = NuclearNorm < Frechet < ProjNorm < AgreeScore

W4: Grammar should be thoroughly checked.

We will thoroughly check the grammar of the entire article based on your suggestions.

Q4: Missing a related work

We will add a description of this study [9] in the related work section: “Chen et al. (2022) propose the SEES framework to estimate performance shift by considering the joint shift of both labels and features.”

[1] Devin Guillory et al. Predicting with confidence on unseen distributions. ICCV 2021.

[2] Jiefeng Chen et al. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. Neurips 2021.

[3] Thomas Unterthiner et al. Predicting neural network accuracy from weights. arXiv 2020.

[4] Charles H Martin et al. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications 2021.

[5] Weijian Deng et al. What does rotation prediction tell us about classifier accuracy under varying testing environments? ICML 2021.

[6] Ru Peng et al. CAME: Contrastive automated model evaluation. ICCV 2023.

[7] Weijie Tu et al. A bag-of-prototypes representation for dataset-level applications. CVPR 2023.

[8] Yuzhe Lu et al. Characterizing Out-of-Distribution Error via Optimal Transport, Neurips 2023.

[9] Chen et al. Estimating and explaining model performance when both covariates and labels shift. In NeurIPS 2022

We would like to thank you for your constructive comments, which are very helpful and make our paper even stronger! We sincerely hope that our response addresses your concerns.

W1&Q1: Please clarify the claims on the overconfidence issue, substantial storage, and computation cost

We would like to give a clearer clarification of the advantages of MDE compared to existing AutoEval methods from three aspects:

i. Overconfidence issue:

For methods based on model confidence, such as ConfScore, Entropy, DoC [1], and ATC, we provide an intuitive example to illustrate why they suffer from overconfidence issues.

Given two samples with labels [0, 1, 0] and [0, 1, 0], a model outputs softmax scores for the two samples such as [0.8, 0.1, 0.1] and [0.1, 0.8, 0.1]. At this point, the model's confidence is 80%, while the prediction accuracy is 50%, and the MAE between the two quantities is 30%. The model overestimates the prediction truthiness of its softmax probability, i.e., "overconfidence”.

Therefore, it is necessary to introduce model calibration techniques. We adopt commonly used temperature scaling (i.e. dividing logits by the coefficient $t$ before applying Softmax) in the ATC paper for demonstration:

Let $t$ =2, the two softmax scores will be transformed to [0.4, 0.05, 0.05] and [0.05, 0.4, 0.05]. Now the model's Softmax confidence is 40%, while the prediction accuracy remains at 50%, the MAE will decrease by 20% for accuracy prediction.

However, MDE is a meta-distribution statistical metric based on the sample’s energy (a non-normalized scalar), which does not depend on Softmax confidence and thus is not affected by overconfidence.

ii. Substantial storage and Computation cost:

1. For methods based on prediction disagreement, such as self-training ensembles [2]; and AgreeScore, they generally use the disagreement of the prediction outputs of two or even multiple models for accuracy predictions. They additionally train or fine-tune models and store these models, which incurs additional storage and computing overhead.
2. For methods based on model parameters, such as ProjNorm, first use the model to label the target data, then fine-tune the model on the pseudo data and finally calculate the model parameter shift to predict accuracy, while Weights [3] and Norm-based metrics [4] estimate the accuracy through the norm of the model weight matrix and bias vector. These methods either fine-tune the model or use model parameters to calculate, which is quite a bit of storage burden and computational cost.
3. For methods based on self-supervised tasks, such as Rotation [5] and CAME [6], they jointly train the model from scratch by self-supervised task together with the classification task, and use the self-supervised accuracy as a proxy for classification accuracy. This way of retraining the model inevitably introduces extra computational overhead.
4. For methods based on dataset distribution shift, such as Frechet and Bag-of-prototypes [7], when predicting the model accuracy of the target dataset, they need to access the training set, obtain model features, and calculate the mean of feature or cluster the features, this sort of method also consumes computing resources in practice.

However, MDE only requires the model logits on the target dataset for calculation, without the above highlighted constraints. This effectively reduces the substantial storage and computation cost.

iii. Compared with ATC, MDE requires training a linear regression model

1. The buildup of fitting regression models via strong correlations between unsupervised indicator and accuracy, which is just a common workflow for several published AutoEval works ([4], [5], ProjNorm, Nuclear Norm, Frechet, Bag-of-prototypes, etc), no exclusive to MDE. In practice, we only call the sklearn.linear_model.LinearRegressionto fit the regression model, which is almost not time-consuming.
2. The ATC method requires identifying a threshold on the source validation data and performing model calibration to obtain better accuracy predictions, it also has its own computational overhead.
3. Most importantly, the performance of MDE significantly surpasses ATC. Compared with ATC, the mean absolute error of MDE dropped a lot (CIFAR-10: 8.15→1.78, TinyImageNet: 11.37→2.15, ImageNet: 11.51→5.48, MNLI: 15.77->5.76, Wilds: 6.35→2.26), the correlation also improves especially for MNLI ($\rho$: 0.647→0.850, $R^2$: 0.459→0.680).

W2: Please clearly explain the contributions of this paper.

We respectfully disagree with the comment that our work has limited contributions, the following statements are intended to dispel the concerns raised by the reviewers:

i. The use of Energy score as a replacement for Softmax score：

1. MDE and Softmax score are different in essence and usage.

• Essence: MDE is defined as the meta-distribution statistic of non-normalized energy at the dataset level, while Softmax score is the maximum value of the normalized logit vector for a single sample.

• Usage: Softmax score typically reflects the classification accuracy through measures such as the mean, mean difference, or the data proportion below a certain threshold. On the other hand, MDE predicts accuracy by a regression model.

2. MDE and Softmax score have different mathematical forms.

$

\mathcal{MDE}(x;f) =-\frac{1}{|N|} \sum_{i=1}^N \log \operatorname{Softmax} E(x;f) =-\frac{1}{|N|} \sum_{i=1}^N \log \frac{e^{E\left(x_n;f\right)}}{\sum_{i=1}^N e^{E\left(x_i;f\right)}},

$

$

\max _y p(y \mid \mathbf{x})=\max _y \frac{e^{f_y(\mathbf{x})}}{\sum_i e^{f_i(\mathbf{x})}}=\frac{e^{f^{\max }(\mathbf{x})}}{\sum_i e^{f_i(\mathbf{x})}}

$

3. MDE is more suitable for accuracy prediction than Softmax score

   To demonstrate this, we decompose the softmax confidence by logarithmizing it as follows:

$

\begin{aligned}
\log \max _y p(y \mid \mathbf{x}) & =E\left(\mathbf{x} ; f(\mathbf{x})-f^{\max }(\mathbf{x})\right) \\\\
& =E(\mathbf{x} ; f)+f^{\max }(\mathbf{x})
\end{aligned}

$

 Then, we find $f^{max}(x)$ tends to be lower and $E(x; f)$ tends to be higher for OOD data, and vice versa. This shifting results in Softmax confidence that is no longer suitable for accuracy prediction, while MDE is not affected by this bothersome issue.

ii. Existing work has discussed the connection between OOD detection and AutoEval, and this paper does not extend this scope.

1. This paper does not explore the relationship between OOD detection and AutoEval but rather proposes a novel AutoEval method.

2. AutoEval has fundamental differences with OOD detection. OOD detection is to identify outlier test samples that are different from training distributions, but AutoEval is towards unsupervised estimating of the model’s testing accuracy.

iii. Reviewers Dsfd and 2CiW both agree that the MDE method has theoretical interpretability and practical application effectiveness. Reviewer 2CiW also acknowledged the relatively high novelty of the MDE method.

W3: Unconvincing theoretical analysis

We respectfully disagree with this comment. Considering the scenario mentioned by the reviewer where the model accurately classifies images with lower softmax scores, we provide the following example to illustrate how the Eq.(9) of Theorem 3.1 substantiates the MDE can work in such a case.

$

\Delta^{i}=\mathcal{MDE}^i - \mathcal{L}^i_{\text {nll }} = f_{y_i}({x_i})/T - \max _{j \in \mathcal{Y}}{f_j\left(x_i\right)/T} =\begin{cases}0, & \text { if } j=y_i, \\ <0, & \text { if } j \neq y_i,\end{cases}, \quad\quad(9)

$

Given a sample with the label [0, 1, 0], the model outputs the Softmax score for this sample as [0.4, 0.5, 0.1]. According to Theorem 3.1, we know that $y_i$ =2, $j$ corresponding to $\max _{j \in \mathcal{Y}}{f_j\left(x_i\right)/T}$ is 2, so the model classifies correctly with a lower softmax score.

In turn, when the model incorrectly classifies images with higher softmax scores, that is given a sample with the label [0, 1, 0], the model outputs the Softmax score for this sample as [0.5, 0.4, 0.1]. According to Theorem 3.1, $y_i$ is still 2, but $j$ becomes 1, so $j$ is not equal to $y_i$, and $f_{y_i}\left(x_i\right) / T-\max _{j \in \mathcal{Y}} f_j\left(x_i\right) / T = 0.4 - 0.5$ < 0.

From the above examples, by judging whether the term of Eq. 9 is less than 0, we can know whether the index $j$ belongs to the maximum logits equal to the label $y$, and then we can obtain the accuracy of the classifier $f$. Thus, we mathematically establish a connection between MDE and classification risk.

Moreover, both Reviewers Dsfd and 2CiW concurred that the theoretical proof of this paper has substantiated the effectiveness of the proposed method.

W4&Q2&Q3: Please report the results on the ImageNet-1k setup and compare MDE with existing works

Following the reviewer’s advice, we add two tables that compare the correlation and MAE with existing methods (including COT [8]) on the ImageNet-1K setup.

Table 1. Correlation comparison with existing methods on ImageNet Setup

Table 2. Mean Absolute Error (MAE) comparison with existing methods on ImageNet Setup

In summary, MDE demonstrates significant performance gains over existing AutoEval methods (including COT) on ImageNet-1K setup, further confirming the effectiveness of MDE.

Some remarks:

1. The COT method requires pre-calibration of the classifier, which is not a constraint in MDE.

2. The COT paper was accepted in Neurips 2023 on October 27, while the submission deadline for ICLR 2024 is on September 28. According to the paper submission principles, this is a paper of the same period and does not require comparison.

3. To make the experiment more convincing, we fairly compared MDE and COT under the same experimental setting. It can be seen that without calibrating the classifier, the performance of MDE is significantly better than COT.

4. Within such a short rebuttal period, we do not wish to engage in a dispute over which one is better. Considering that MDE and COT belong to different technical systems and do not have an incremental innovation relationship, our innovation lies in the domain of energy score systems.

W5: Limited Dataset Diversity

We follow the reviewer's recommendation to report the result on the Wilds datasets with more natural shifts, the experimental setting kept the same as the ATC paper:

Table 3. Mean Absolute Error (MAE) comparison with existing methods on Wilds Setup

The MAE of MDE in predicting OOD test accuracy on various Wilds datasets is optimal again.

By the way, the Breeds dataset is created by more finely stratifying the class structure of ImageNet. Since we have completed the experiment on ImageNet, we have not reported on the Breeds dataset.

Q4: Please provide experiments on OOD

• If the reviewer is referring to "AutoEval problem inspired by the large variation in model accuracy on the ood dataset, but we did not provide the experiments on ood datasets":

  Actually, all experiments in the paper were conducted on OOD. We assume that this misunderstanding arises from the lack of explicit indication in the captions of the experimental figures/tables that the dataset is actually OOD.

  The correlations for each dataset in Table 1 are actually computed on their corresponding synthetic sets, involving the application of synthetic shift to the source dataset. For instance, the correlation for CIFAR-10 specifically refers to the $\rho$, $R^2$ calculated on CIFAR-10-C.

  In Table 4, we calculate the MAE on an unseen test set with natural shifts (i.e., an OOD dataset) in comparison to the source data. For instance, for the CIFAR-10 dataset, we predict the model‘s accuracy on the CINIC-10 OOD dataset.

• If the reviewer intends to convey that "OOD detection has inspired the AutoEval problem, but we did not provide experiments on ood detection”:

  i. AutoEval and OOD detection cannot be directly compared as they are two entirely different problems. Among them, OOD detection is to identify outlier test samples that are different from training distributions, but AutoEval is towards unsupervised estimating of the model’s testing accuracy.

  ii. This paper was not inspired by OOD detection techniques, but rather by the characteristics of the energy that fulfilled our desire to build an efficient and effective AutoEval framework.

[1] Mayee Chen et al. Mandoline: Model evaluation under distribution shift. ICML 2021.

[2] Simona Maggio et al. Performance prediction under dataset shift. ICPR 2022.

[3] Yoo, Donggeun et al. Learning loss for active learning. CVPR 2019.

[4] Schelter, Sebastian et al. Learning to validate the predictions of black box classifiers on unseen data. SIGMOD 2020.

[5] Emmanouil Antonios Platanios et al. Estimating accuracy from unlabeled data: A bayesian approach. ICML 2016.

[6] Licong Guan et al. Instance segmentation model evaluation and rapid deployment for autonomous driving using domain differences. IEEE Transactions on Intelligent Transportation Systems, 2023.

[7] Yujie Lu et al. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Neurips 2023.

[8] Yue X et al. Automatic evaluation of attribution by large language models. EMNLP 2023 Findings.

[9] Kuhn et. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. ICLR 2023.

[10] C. H. Martin. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 2021.

[11] Ciprian A Corneanu et al. Computing the testing error without a testing set. CVPR, 2021.

[12] Dan Hendrycks et al. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR 2017.

[13] Devin Guillory et al. Predicting with confidence on unseen distributions. CVPR 2021.

[14] Saurabh Garg et al. Leveraging unlabeled data to predict out-of-distribution performance. ICLR 2022.

[15] Weijian Deng et al. Characterizingterising prediction matrix for unsupervised accuracy estimation. ICML 2023.

We would like to thanks for your valuable feedback and far-sighted insights. We hope the following statements will provide you gain an in-depth understanding of AutoEval and our MDE method, thereby dispelling your concerns.

Q1: A clear restatement of AutoEval Problem

i. What is AutoEval?

AutoEval task is to evaluate the performance of a trained model on datasets without accessing the real labels.

Specifically, given a trained model $f$ and an unlabeled OOD dataset $D_u=\lbrace x_i \rbrace_{i=1}^N$ with $N$ samples, AutoEval aims to predict the accuracy of $f$ on $D_u$.

The common AutoEval workflow is to develop an indicator that is correlated to accuracy but to calculate without requiring labels.

The AutoEval method is assessed by the correlation coefficient between the indicator and the accuracy, as well as the MAE between the predicted accuracy and ground-truth accuracy on OOD datasets.

ii. Why is AutoEval needed?

1. The AutoEval task is long-standing and has been studied in a wide range of areas, e.g. text classification [1], structured data [2], active learning [3], database [4], medicine images [5], self-driving images [6], AIGC [7], LLM [9,9], etc.

2. The statistics for these AutoEval papers appearing in the top-tier publications: Nature Communications(1, [10]), ICML(9), NeurIPS(6), ICLR(4), JMLR(2), AISTATS(1), CVPR(4, including the CVPR2020 best paper nominations [11]), ICCV(2), ACL(1), EMNLP(2), ECAL(1), SIGMOD(1), ICSE(1), TNNLS(1), a total of 36 paper.

3. Also, reviewers Dsfd and 2CiW both agreed in Strengths 1 of their reviews that AutoEval is valuable in the real-world deployment of DNNs.

Q2: How does AutoEval differ from uncertainty estimation?

Unlike uncertainty estimation to estimate the confidence of model predictions to convey information about when a model’s output should (or should not) be trusted, AutoEval directly predicts the accuracy of model output.

In this regard, AutoEval is a task that assesses the effectiveness and deployment worthiness of a model by directly predicting its accuracy in a testing environment.

Q3: Unlike common methods in uncertainty estimation, MDE is not an incremental approach.

i. Using logits is a general operation:

Notably, AutoEval methods based on model outputs (e.g. ConfScore [12], Entropy [13], ATC [14], NuclearNorm [15], etc.) all utilize logit, because it is closely associated with model accuracy.

So MDE just adopted this conventional practice and has achieved remarkable performance.

ii. MDE are totally different from common methods in uncertainty estimation

Since the reviewer did not clarify whether MDE is similar to which methods in uncertainty estimation, we compare MDE with the widely used Softmax confidence:

1. Comparing the essence of the two methods, MDE is the meta-distribution statistic of non-normalized energy at the dataset level, while Softmax confidence is the maximum value of the normalized logit vector for a single sample.

2. Comparing the mathematical form of the two methods, they are completely different:

$

\mathcal{MDE}(x;f) =-\frac{1}{|N|} \sum_{i=1}^N \log \operatorname{Softmax} E(x;f) =-\frac{1}{|N|} \sum_{i=1}^N \log 
\frac{e^{E\left(x_n;f\right)}}{\sum_{i=1}^N e^{E\left(x_i;f\right)}}, 

$

$

\max _y p(y \mid \mathbf{x})=\max _y \frac{e^{f_y(\mathbf{x})}}{\sum_i e^{f_i(\mathbf{x})}}=\frac{e^{f^{\max }(\mathbf{x})}}{\sum_i e^{f_i(\mathbf{x})}}

$

3. Compare the two methods for accuracy prediction on OOD data, we decompose the softmax confidence by logarithmizing it as follows:

$

\begin{aligned}
\log \max _y p(y \mid \mathbf{x}) & =E\left(\mathbf{x} ; f(\mathbf{x})-f^{\max }(\mathbf{x})\right) \\\\
& =E(\mathbf{x} ; f)+f^{\max }(\mathbf{x})
\end{aligned}

$

Then, we find $f^{max}(x)$ tends to be lower and $E(x; f)$ tends to be higher for OOD data, and vice versa. This shifting results in Softmax confidence that is no longer suitable for accuracy prediction, while MDE is not affected by this bothersome issue.

iii. Reviewer 2CiW also acknowledged the relatively high novelty of the MDE method in Strengths 2 of his/her review.

Q7: Is there any comparison on the evaluation of domain-shift data with real labels?

Yes, we have conducted all experiments of this work on the domain-shift dataset. We assume that this misunderstanding arises from the lack of explicit indication in the captions of the experimental figures/tables that the dataset is actually OOD.

The correlation for each dataset in Table 1 is actually computed on its corresponding synthetic sets, which is a synthetic domain-shifted dataset for the source dataset,

In Table 4, we calculate the MAE on the unseen test sets, which are natural domain-shifted datasets in comparison to the source dataset. For instance, for the CIFAR-10 dataset, we evaluate the correlation on the CIFAR-10-C synthetic domain-shifted dataset and calculate the MAE on natural domain-shifted datasets such as CINIC-10, etc.

Notably, the MDE method does not require access to the real labels when predicting the model accuracy on domain-shift data. If we understand the reviewer right, we do need the real labels to provide real accuracy when evaluating the MAE of the MDE method.

W5&Q5: Small range temperature ablation and how to determine the best temperature

The ablation results for a wide range of temperature hyperparameters ($T$ from 0.1 to 100) are shown below:

In fact, despite the ideal case for MDE being $T \rightarrow 0$ in Theorem 3.1, our empirical results found that the performance variation caused by $T$ from 0 to 1 is minimal. Thus, we fix $T = 1$ to ensure that the classification accuracy is not affected so that we can compare fairly with baselines. Most importantly, we did not rely on T to tune performance.

W1&W2&W7: Non-standard symbol format and missing EBM citations.

Thanks for the reminders! We will complete these symbols and references.

Q1: A clerical error about MDE.

Sorry, we missed a summation sign after the $-\frac{1}{|N|}$ please excuse this clerical error.

Eq.(5) is an expectation term：$\mathcal{MDE}(x;f) =-\frac{1}{|N|} \sum_{i=1}^N \log \operatorname{Softmax} E(x;f) =-\frac{1}{|N|} \sum_{i=1}^N \log \frac{e^{E\left(x_n;f\right)}}{\sum_{i=1}^N e^{E\left(x_i;f\right)}}.$

Q2&Q4: How to generate the synthesized set for regression and evaluate correlation in Table 1?

We take the CIFAR-10 setup as an example to clarify this doubt. At this setup, the synthetic set is CIFAR-10-C datasets, which involve 19 types of corruption with 5 different intensity levels applied to the CIFAR-10 validation set. When encountering a new data setup, we can perform the same corruption operation on its validation set to obtain the synthetic set. The correlation score of each dataset in Table 1 refers to the coefficients $\rho$, $R^2$ calculated on the <MDE, accuracy> pair of its corresponding synthesized set.

Q3: How to correct performance drop on new datasets？

• If the reviewer wants to know “how to mitigate the MAE and correlation degradation of the MDE method on OOD datasets”:

  All AutoEval methods may suffer from performance degradation on unknown new datasets, and this is not exclusive to MDE alone.

  This performance decline typically arises from severe domain shifts between the new dataset and the training dataset, such as adversarial perturbations and extreme class imbalances. Faced with these challenging scenes, how to sample the new dataset to capture its main shift types and combine it into the synthetic set may be a promising solution. We sincerely request the community to actively involve themselves in addressing this issue.

• If the reviewer wants to know “how to alleviate the accuracy decline of models on OOD datasets”:

  Once MDE finds that the model performance drops significantly on the new dataset, we can introduce related technologies of domain adaptation or domain generalization to alleviate this problem, but this is not the scope of this article.

Q6: Comparing the performance of different regression models.

The choice of different regression models does not affect the accuracy prediction performance. With a fixed dataset (CIFAR-10) and backbone (VGG11), we compare the correlation and MAE results between RobustLinearRegression and LinearRegression below:

We would like to thanks for your thoughtful and insightful reply! We sincerely hope our response can resolve your concerns.

W3: Unclear relationship between MDE and EBM.

A standard Energy-Based Model (EBM) is trained with implicit MCMC sampling [1,2] to learn an energy function that assigns low energy values to input x in the sampled data distribution.

The proposed MDE is not an EBM, but an indicator calculated with resort to the energy function of EBM. Importantly, the MDE indicator can predict the model performance on unlabeled OOD datasets in real-world scenarios, achieving the SOTA benchmark without requiring much extra overhead.

In the original paper, we first introduce that the classifier can be regarded as an EBM (Eq.1~Eq. 3), and then we calculate the MDE indicator through the "energy" of all data points. (Eq.4).

That is the relationship between MDE and EBM.

W4&W6: Transformations such as shear etc. is easy to synthesize, and requires more complicated real dataset for evaluation

As suggested by the reviewer, to substantiate the feasibility of MDE in the wild, we compare MDE with other strong baselines on the more complicated real datasets — ImageNet and Wilds, where Wilds includes the MRI-like medical image dataset Camelyon17.

Table 1. Correlation comparison with existing methods on ImageNet Setup

Table 2. Mean Absolute Error (MAE) comparison with existing methods on ImageNet Setup

Table 3. Mean Absolute Error (MAE) comparison with existing methods on Wilds Setup

The considerable performance on both ImageNet and Wilds reflects the versatility of MDE. In the future, we envision deploying the MDE into broader real-world domains to demonstrate its scalability.

[1] Yilun Du et al. Implicit generation and modeling with energy based models. Neurips 2019.

[2] Will Grathwohl, et al. Your classifier is secretly an energy based model and you should treat it like one. ICLR 2020.