OmniInput: A Model-centric Evaluation Framework through Output Distribution

Q1:

1. Estimate is provided in caption of Figure 4 with 40 bins, each containing at least 300 samples (at least 12000 samples). GWL has a gradient component to encourage the sampler to visit under-visited output values and it does not need many data points to ensure 50 samples.
2. The scale of log(recall) in Figs. 2 and 4 is in natural log scale.
3. (W2 and Q1.3) Yes, we believe our approach can be applied to problems with larger scales such as ImageNet classification directly. Please also refer to the general discussion at the beginning of the response "Common concerns on representative dataset and multi-class classification" about the scalability where we discuss how to interpret our results in the experimental section for other datasets and how to extend our method to multi-class classification (applied for CIFAR100). Additional results of CIFAR100 are presented.

Q2: Yes. Our method introduces a paradigm shift in the methodology for evaluating models, suggesting a novel approach that prioritizes a model-centric perspective over a data-centric one. Unlike the traditional practice of hypothesizing potential model failures then conducting experiments for validation, our model-centric approach aims to comprehensively understand a model's intrinsic biases, rather than relying on artificially constructed datasets to assess model performance. By doing so, our method mitigates dataset biases, with improved efficiency resulting as a byproduct, as visual diversity is notably limited across various models. In the context of adversarial attacks, it is noteworthy that existing models, particularly classifiers, often lack the capability to propose informative samples. We leave this research focusing on the generation of informative samples to gain a model-centric understanding of adversarial samples as future work.

Thank you for your valuable comments and feedback! We replied by adding the details to OmniInput (number of samples and scale of recall). We added experiments for representative datasets (CIFAR100 and CIFAR10) with multi-class classification for the larger-scale question. We also clarified the advantage of our approach besides efficiency. Could you please check and see if our revision will make our work meet the ICLR bar? Please let us know if you have any questions. Really appreciate that!

Thank you for your comments. The metric applied in the entire input space inflects that we can trust a model that have larger AUPR compared to the ones with smaller AUPR, assuming we do not know what could be the input the model. This is practical since in the real world, especially in out-of-distribution detection, if we cannot guarantee what the users will input the model. If we test on a predefined set of datasets but the users tend to use the model in an unexpected way, we need such a metric to compare models to know what model can trust more than the others. For example, we might have checked the model can detect Gaussian noise, but what if we have a different Gaussian noise of input with different mean and variance or uniform distribution of the noise as input? We cannot enumerate all the possibility, and the value of our work is we can still estimate the errors a model can make in the most general testing set (entire input space).

Thus, the model with higher precision is expected to be less vulnerable to overconfident prediction, and a model with higher AUPR is expected to approach closer to the desired input distribution of training (testing) set. Our work does not target specifically for adversarial attacks at this moment which requires modification to focus on data (informative samples) distribution.

(W1) Our setting does not assume we know how the model is trained. Thus whether the model simply memorizes the training set (overfitting) does affect our methodology. Overfitting only affects the results we are about to analyze to understand model performance.

(W2, Q1) We discussed how to extend the framework to multi-category settings and more advanced methods of sampling in “Common concerns on representative dataset and multi-class classification” above in the general discussion. To verify the scalability of multi-classes and larger inputs (CIFAR10 and CIFAR100), we add a new Section I in the Appendix with a detailed discussion of the output distribution and representative samples.

Thank you for your valuable comments and feedback! We added experiments for representative datasets (CIFAR100 and CIFAR10 with multi-class classification). We also replied to the scalability problem of sampling for larger input (though this is not the focus of OmniInput), and about the overfitting concern. Could you please check and see if our revision will make our work meet the ICLR bar? Please let us know if you have any questions. Really appreciate that!

(W1) The major technical contribution of this work is a novel approach to quantify model performance across the entire input space, as opposed to a novel sampler. This work is enabled by a good sampler, but the framework proposed in this work is completely new, and its implication transcends a mere application of the sampler. Specifically, the new contributions of this work include:

1. Model-centric performance evaluation: different from the conventional practice of hypothesizing potential model failures and subsequently conducting experiments for validation, our methodology revolves around the concept of model-centric evaluations and samples directly from the model under evaluation. The objective is to comprehend the intrinsic biases of the model self-sufficiently, eschewing the conventional approach of relying on human-probed datasets or other models to estimate model performance. This methodological renovation marks a substantial difference from prior works in the field of model evaluation.
2. Model performance available for the entire input space: the utilization of the model’s output distribution represents an advancement of performance evaluation that extends from representative samples (only for generative models) to the entire input space (for both generative models and classifiers). This is the first time where model performance is estimated for the entire input space of a dataset.
3. Ability to quantify overconfident predictions: the issue of overconfident predictions has been a longstanding concern in out-of-distribution (OOD) detection. Our contribution is that we are the first to quantitatively assess overconfident predictions by harnessing the output distribution. This novel approach allows for a quantitative comparison between two models, even when overconfident predictions are observed in both models. This is a significant advance in the context of OOD detection. Our framework not only provides a viable approach to quantify overconfident predictions; more importantly, it provides a comprehensive understanding of models’ performance even when overconfident predictions exist for both models.

(W2) We extended the validation of our framework to CIFAR-100 to provide more evidence that our framework is generic to different applications. Please also refer to “Common concerns on representative dataset and multi-class classification.” To verify the scalability of multi-classes and larger inputs (CIFAR10 and CIFAR100), we add a new Section I in the Appendix with detailed discussion of the output distribution and representative samples.

Thank you for your valuable comments and feedback! We added experiments for representative datasets (CIFAR100 and CIFAR10) with multi-class classification. We also clarified our contribution compared to the previous works. Could you please check and see if our revision will make our work meet the ICLR bar? Please let us know if you have any questions. Really appreciate that!

Thank you for your comment and suggestion.

We expect the safety issues to be able to deal with unexpectedness of the testing cases, and our framework is able estimate the errors in these applications. While it is hard to say how likely "a car drawn in sketch" will appear in the real world application, the model should be able to handle obvious issues such as uninformative noise. This can happen as glitch deal to hardware failures when autonomous car is deployed. If we are confident that the typical "styles" appear often in the applications and the unexpected glitch won't lead to fatal accidents, we can just test on what we believe the model might fail. If that's the application and we can model these typical scenarios using generative models etc, our framework which uses propose samples and output distribution to compute precision and recall still works. The only difference is the entire input space becomes the data space proposed by the generative model.

We thank the reviewer's recommendation on the test sets. Testing domain generalization datasets will limit our conclusions on those dataset samples, similar to the data space proposed by the generative models and the conclusions will become limited to typical precision/recall on a predefined dataset. Would you like to elaborate what type of conclusions you expect when we test those datasets, as our setting and their setting both seem be "agnostic" but there is a signficant motivational difference?

Thank you again for your comment and suggestions.

(W1) This is a good suggestion. We extended the validation of our framework to CIFAR-10 (and SST2) to provide more evidence that our framework is generic to different ML models. Please also refer to the "Common concerns on representative dataset and multi-class classification" for how to interpret our results in the experimental section for other datasets, as well as how our method is extended to multi-class classification (applied to CIFAR100). Additional results of CIFAR100 are presented in Section I in the appendix.

(W2) The generalization to state-of-the-art vision models amounts to scaling to large models, which is linearly proportional to the number of parameters, as both forward and backward functions are called during the sampling process. The ViT model and its variants are transformer-based models, which we expect to have similarities to the transformer models we have tested in the NLP application. We believe our framework would be applicable to other models with similar architectures as we treat the middle layers as a complete black-box.

(W3) We added the following discussion in the “Related works” section in the manuscript. These previous studies have similarly used generated samples for model evaluation and [3] also asserts the ability to assess models without reliance on a designated test set. The rationale of our work is based on a similar model-centric principle; however, our work brings about several novel contributions to existing work:

1. Our primary objective is to estimate precisions and recalls for the entire input space instead of only deriving the performance based on representative samples. This necessitates an unbiased selection of samples that are mapped to the same output value by the model to be evaluated, and the output distribution for generalization. In contrast to prior works which perturb attributes to expand samples beyond designated test sets, our study is different in two key aspects: first, the cited papers did not investigate the distribution of representative samples mapped to the same output in the entire input space; second, they did not recognize the implication of the output distribution for evaluating model performance.
2. Our approach is different from a perturb-and-evaluate framework, wherein sample generation is the major objective. On the contrary, our emphasis lies in the identification of major biases that models may potentially harbor, thereby gaining an understanding of the tasks that the models might perform inadequately. This transcends the narrow confines of scrutinizing isolated properties or attributes in other existing works.
3. Our approach goes beyond established methods and settings: we are the first to advocate the consideration of output distribution of a ML model for assessing its performance and gaining a comprehensive understanding and quantification of the predominant biases inherent in a model. As the intrinsic complexity of our new method far surpasses conventional 'perturb-and-evaluate' methods, we made a deliberate choice to compromise scalability in the current manuscript for the sake of establishing a proof-of-principle of the capability and implication of our framework. Nevertheless, the methodology is generic, generalizable, and scalable to other scenarios when computational resources permit.
4. All the four cited papers used other generators to generate samples for evaluating a model. On the contrary, we used a sampler to sample the model to be evaluated. Sampling is transparent with convergence estimates, but other generators are still considered as black boxes. Given the inherently unknown biases in models such as CLIP[3], StyleGAN[2], LLM[4], and generative models [1,4], utilizing other models to evaluate a model (as explained in the Discussions section in our manuscript) carries the risk of yielding unfair and potentially incorrect conclusions. Our method brings the focus back to the model to be tested, tasking it with generating samples by itself for scrutiny, rather than relying on external agents such as human or other models to come up with testing data. An additional benefit is that this approach offers a novel framework for estimating errors in the entire input space when comparing different models.

Thank you for your reply. Our evaluation framework focuses on the settings where we cannot guarantee any input distribution when the model is deployed. This is extremely important for model safety in the open-world setting where we do not have prior knowledge of what usage cases we might encounter that have not yet been tested. Assuming the data to be tested coming from some input distribution modeled by a generative model limits the generalization of our conclusion. Even if the task is specified and task-specific data distribution is modeled by a generative model, our framework won't change. What is going to change is the sampler that needs to sample according to that distribution, but this is future work not the focus of our framework.

Lastly, previous work [1] has shown even (modern?) generative models like VAE, pixelCNN or flow-based models are not (completely) capable of modeling data distribution, “density learned by flow-based models, VAEs, and PixelCNNs cannot distinguish images of common objects such as dogs, trucks, and horses (i.e. CIFAR-10) from those of house numbers (i.e. SVHN), assigning a higher likelihood to the latter when the model is trained on the former.”

[1] Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., & Lakshminarayanan, B. (2018). Do deep generative models know what they don’t know?. arXiv preprint arXiv:1810.09136.