Towards Accurate Deep Learning Model Selection: A Calibrated Metric Approach

1. To my understanding, biased mean predictions happen when examples are infinite. However, since the validation set (specifically $D_{val-bias}$ is typically smaller than the training set, why can we expect bias correction with $D_{val-bias}$ is more reliable than the training set when applying the model to unseen test data?

2. It appears that the experiments assume Pipeline A should outperform Pipeline B, but I found no clear explanation supporting this assumption in Tables 3, 5, 6, and 7. Without justification, it's difficult to interpret the effectiveness of the metrics in distinguishing between the pipelines.

3. Does the proposed method support multiclass classification problems? For binary classification, is this method particularly suitable for CTR or generally applicable to other datasets?

4. The term “calibrated” here is a bit ambiguous as it usually refers to confidence calibration in the context of deep learning. Clarifying the exact meaning will be helpful.

The main weakness I see with this work is the details of implementing this framework in arbitrary settings, and whether the justifications in the paper for the framework actually translate to such settings. My worry is that this paper essentially produces a modification to standard loss functions for evaluating testing performance, but we do not know in practice if this modification is actually getting better at measuring the true attribute we care about (hierarchy of models). Additionally, I am also concerned by the lack of comparisons to more recent related work.

Framework Details: Algorithm 1, in my view, is too high-level to be immediately used for the very general setting this paper tries to cover. In particular, the key aspect of this algorithm is the bias correction function $f_e$, but there is no procedure or even intuition for how one should think about this function. The examples regarding log loss and MSE are very helpful and make sense -- we just introduce a single parameter that adjusts the logits/outputs, but at the very least this notion deserves more explicit exposition at the start of Section 3. Another important aspect is how to actually choose the split of the validation data for fitting the bias correction vs. evaluation; the paper uses 2% for fitting and the remaining 18% for evaluation but this seems arbitrary. A discussion of the trade-off here would be useful. Lastly, as more of a pedantic point, in the experiments it appears that val data corresponds to test data, so it would also be useful to clarify terminology.

Applicability in Practice: The applicability of the proposed framework rests on the idea that the calibrated metric $\bar{e}$ has the same behavior as the original metric $e$ (in the sense that they order different models the same way when taking into account the ground truth distribution) but lower estimation variance. However, there is no justification that this is actually the case in the practical verifications; the theory in the paper applies only to linear regression with jointly Gaussian data, which is very much a toy setting. The experiments compare the "accuracy" (in the sense defined in the paper) of the calibrated metric $\bar{e}$ to that of the original metric $e$, but this comparison only makes sense if we know a priori that one modeling pipeline is better than another (in which case we are checking whether $\bar{e}$ uncovers the better model more accurately than $e$). I understand that intuitively we do have a sense of which pipeline should be better (since we a priori know that some features/training techniques make a nontrivial difference in the settings being considered), but given the generality of the proposed framework I don't see how the comparisons between $\bar{e}$ and $e$ can be conducted in arbitrary settings. For example, we might get better "accuracy" with \bar{e} in some setting but it is actually picking out the worse model (from the perspective of the test distribution).

Related Work: This paper has a very limited comparison to recent related work, which is concerning given that there has been a decent amount of work in this direction over the past few years. At the very least, this paper should compare to the ideas in [1] and the related work cited therein, as [1] also analyzes the variance of metrics computed on held-out test data and what conclusions can be drawn from comparing test metrics across different random training runs.

In summary, while I find the ideas in the paper to be interesting and the problem being considered to be important, I think stronger justifications (and details) are necessary to justify acceptance of the proposed method so I lean reject (falling somewhere between a 3 and a 5). I am happy to update my score upon engaging with the authors if they can clear up some of the issues pointed out above.

[1] Jordan, Keller. “On the Variance of Neural Network Training with respect to Test Sets and Distributions.” International Conference on Learning Representations (2023).

• The proposed calibrated loss is theoretically supported only under the assumption of linear regression. It is unclear whether the proposed loss is theoretically valid with other models, such as deep neural networks or decision trees.
• The experiments compared the non-calibrated and calibrated losses, but the final performance, i.e., standard accuracy evaluated by the 0-1 loss, is not reported. Thus, it is unclear if the model selected by the proposed calibrated loss will be the best.
• The comparison with the other model selection methods and/or the discussion about related work are missing. For example, the method can be compared with cross-validation and information criteria from the viewpoint of the model selection.

• The work is not properly put in context of the research on deep learning model calibration. The Related Works section is extremely light, overly generic to the point of being barely relevant, and oblivious of research on model calibration, when the paper is essentially leveraging a basic calibration technique. See e.g. [1,2,3] as starting pointers. Relatedly, the recalibration technique proposed for binary classification (Eq 6) is given ad-hoc without a reference.

• The theoretical analysis is very limited (linear regression only with joint Gaussian assumption) and somewhat superficial. It assumes perfect knowledge of target E[Y] and does not account for the fact that we have a finite validation set. Finite validation induces a noisy recalibration, moreover having sacrificed part of the set for calibration will yield additional variance in the estimation of the expected risk. None of these effects are accounted for.

• I find the work oversells basic recalibration as a new metric, instead of focusing on the core problems with miscalibration: why miscalibrated models shouldn’t be used for model selection (the core of the work’s contribution), and why we learn such miscalibrated models in the first place.

• Limited scope of the exposition (to binary classification and regression) and empirical evaluation. The approach is only relevant to the extent that the training pipeline produces miscalibrated models (in the limited sense of failing to match target averages). It would be more convincing if the authors had shown that the problem and remedy are relevant more broadly, beyond the two specific tasks studied, s.a. including established deep learning multiclass image classification benchmarks and pipelines.

This paper has several weaknesses:

• I wonder about the motivation of this problem. As far as I understand, the model $h$ in Alg.1 is trained before the alg.1. If it is the case, why did we need a new metric for pipeline comparison since the naive baseline is compare the accuracy on the validation/testing dataset.
• The theoretical result is only for linear regression, which seems trivial in the current practical setting where over-parameterized models are prevalent.
• This corrected bias is based on the choice of validation set. In the case of datasets where the validation set is not provided in advance, randomly selected $\mathcal D_{val-bias}$ may cause variance in results. Therefore, I wonder about the robustness of this metric with different selected validation sets.
• Why were all pairs of models included in Table 3?
• For related work, there is another line of work called "transferability estimation" which aims to compare different checkpoints/tasks for a specific dataset. Therefore, the literature review would be more comprehensive if authors compared with this line of work.