LaSCal: Label-Shift Calibration without target labels

Can you clearly state the novelty compared to covariate shift methods?

Please see our general response for the perceived lack of novelty. To summarize:

• This paper introduces the first label-free, consistent estimator for target calibration error under label shift. Estimating calibration error under covariate and label-shift are two very different problems.
• The re-weighting strategy in label shift is based on the label distribution ratio, which is fundamentally different from the feature distribution ratio used in covariate shift methods.
• The paper provides both a strong theoretical justification and a thorough empirical analysis, demonstrating the effectiveness of our approach, and showcasing that existing covariate shift methods are suboptimal under label shift (see Table 2). This result emphasizes the necessity for designing recalibration methods tailored to label shift conditions.

Thank you for the valuable feedback and suggestions. We implemented these baselines and performed additional experiments, which we will include in the camera-ready version. We address each question below:

1. Add at least one more relevant baseline for model calibration under label shift

We compared our method both with CPMCN [1] and EM-BCTS [2], on CIFAR-10/100, Amazon, and iWildCam.

CIFAR10-LT

CIFAR-100-LT

Amazon

iWildCam

Overall, on all datasets LaSCal either outperforms or performs competitively with both CPMCN [1] and EM-BCTS [2], as per the macro-average CE across models (note for CIFAR-10 and Amazon the error bars overlap for LaSCal and EM-BCTS). LaSCal's gains are prominent for datasets which feature a large(r) number of classes (100 in CIFAR-100 and 20 in iWildCam). We hypothesize this is due to the increased complexity of the optimization process associated with higher-dimensional spaces, which our approach seems to handle more effectively. Aside from the empirical gains, our method has the following advantages compared to these methods: (1) is based on a consistent estimator of target calibration error; (2) it enables unsupervised calibration on the target distribution, whereas CPMCN [1] and EM-BCTS [2] perform the calibration step on a labeled validation set (from source).

2. Calibration error estimator is perceived as data-hungry

While it is true that the estimator requires a sufficient number of data samples (4000 samples in Fig. 3c) to accurately estimate the calibration error (CE), we do not believe that this should be perceived as data-hungry. We do agree that in severely data-scarce settings, this requirement may limit potential applications, and we will discuss this in the revised manuscript. However, please note that the error rate of our estimator is $O(n^{-1/2} + m^{-1/2})$, which is the same as the weight estimation methods (see Lemma1 for the RLLS estimator in Azizzadenesheli et al. [6], and top paragraph on page 8 in Garg et al. [4] for the EM-BCTS method from Alexandari et al. [2]). Therefore, the data requirement is not unique to our method, but rather is common across all weight estimation-based approaches. Further, Fig. 3c also shows that our estimator has a positive bias in scenarios with limited data. This characteristic is preferable as it prevents the false impression that a model is well-calibrated due to insufficient sample size. In essence, our method errs on the side of caution, ensuring reliability even in data-constrained environments.

3. Effect of recalibration step on calibration error estimation?

Thank you for the interesting question. The label shift gap is not affected by the recalibration step, and therefore the obtained weights from the weight estimation methods remain the same before and after calibration. Having said that, we hypothesize that the precision of our estimator would remain the same. To gain insights, we performed additional experiments on CIFAR-10, comparing the precision of estimating CE using our estimator, compared to ground truth (with labels) before and after the re-calibration step with LaSCal. The empirical results we obtained seem to confirm our hypothesis.

We thank the reviewer for the positive review and feedback on our paper. We provide an explanation to the questions below:

1. Some of the baselines use source data for calibration.

While it is true that compared to i.i.d calibration methods, unsupervised calibration methods (LaSCal, HeadToTail, TransCal, CPCS) have access to samples from the unlabeled target distribution in addition to the labeled validation set (from source), the reasons we included these methods are two-fold:

1. To showcase that traditional i.i.d. calibration methods fall short in the case of label shift and highlight the necessity for designing calibration methods specifically for this scenario.

2. To be consistent with the papers on calibration under covariate shift i.e., HeadToTail, TransCal and CPCS, which also include TS, Vector Scaling, Ensemble TS and IROvA.

In the revised manuscript, we propose to replace two of these baselines (Vector scaling and ETS) with the baselines proposed by Reviewer Lv5P in Table 2, and move the VectScal and ETS to Appendix. We will also group the methods or add markers, to make this difference more visible in the tables.

2. What are the potential causes of label shift?

Some of the potential causes of label shift:

• Change in population demographics: e.g. a model trained on mainly adult population (Hospital A) is deployed in a hospital that primarily serves children (Hospital B).
• Unusual events like disease outbreaks or natural disasters: e.g. during a pneumonia outbreak, $p(Y)$ (e.g. flu) might rise, but the symptoms of the disease $p(X| Y)$ (e.g. cough given flu) do not change.
• Geographic changes: e.g. consider wildlife classification model for identifying animal species based on camera trap images (as in our experiments), trained on images from one region with a particular animal species distribution, and deployed in a different region with a different animal speciries distribution.
• Seasonal changes: A model designed to predict seasonal allergies might face label shift if it is trained using data collected during spring, when pollen levels are high and hence allergies are more frequent, and then applied in autumn when allergies are less frequent

3. Does label shift have an impact on model calibration? Label shift between training and validation sets does not seem to be reflected by calibration.

Model calibration is defined with respect to a data distribution. When that data distribution changes (e.g., when facing label shift), the predicted probabilities of a model trained on the source distribution may no longer reflect the true empirical frequencies of the target distribution, leading to poor calibration. For instance, if a model trained on a skewed dataset predicts a high probability for a rare class, this probability might not accurately reflect the true frequency of that class (conditioned on that probability) in the new distribution where that class is more frequent.

Tables 10-16 in the Appendix confirm that label shift does have an impact on model calibration. For example, in Table 16 we can observe a notable increase of the calibration error from source data (CE_s) to the label-shifted target data (CE_t), particularly prominent for the Amazon dataset. Please note that in our experiments, the training and validation sets (CE_s) are sampled from the source distribution, while the test set (CE_t) is sampled from the label-shifted target distribution.

4. Does the result obtained from calibrating the model with a validation set that has data shift negatively affect the results of the training set?

Our proposed post-hoc calibration method is accuracy-preserving, as it performs temperature scaling by optimizing the CE estimator. Since the logits are scaled with a single parameter, temperature scaling maintains the original order of the predictions, ensuring that the predicted class remains unchanged. Therefore, calibration on label-shifted data does not negatively affect the predictive performance of the classifier on the training (source) set.

Thank you for the valuable feedback and future work suggestions. We performed further experiments which serve as an interesting addition to our paper. Regarding the questions/concerns:

1. Proposed method is perceived as straightforward: While our method builds upon the well-known importance weighting technique, our contribution extends beyond the straightforward application of this method.

• We derive the first consistent calibration error estimator under label shift, without using target labels. Furthermore, the estimator has a known error rate $O(n^{-1/2} + m^{-1/2})$ [L140]. Depending on the choice of kernel, the estimator can be differentiable and integrated as an objective in both post-hoc and trainable calibration methods. To the best of our knowledge, no other estimator exists for target calibration error under label shift with the same properties as ours.

• Based on this estimator, we introduce an accuracy-preserving recalibration method. While some of the papers addressing label shift (CPMCN, Wen et al. [1]; EM-BCTS, Alexandari et al. [2]), do incorporate a calibration step, it is performed on validation (source) data, whereas our method performs unsupervised calibration on the target domain.

• We provide a thorough analysis of our method’s properties, demonstrating its robustness across datasets, modalities and severity of shift. We benchmark different weight estimation methods and identify the most robust method for our task. Note that although several weight estimation techniques exist for the label shift scenario, none of them extend their application beyond improving the predictive performance of the classifier.

2. Alternative objective functions and/or metrics: Indeed, our method (LaSCal) naturally extends to the standard ECE. In an additional experiment, we adapted our label-free estimator under label shift to estimate ECE and used that as an objective for temperature scaling. The table below reports ECE (with labels) for several baselines: IID temperature scaling, HeadToTail (as representative for methods derived under covariate shift assumption) and the two new baselines proposed by reviewer Lv5P, which include most recent works (CPMCN [1] @ ICLR 2024) on calibrating models facing label shift. Please see our general response for reasons why we focus on classwise ECE.

Amazon experiments

iWildCam experiments

From the results we see that the observations remain the same as in the main paper: (1) LaSCal significantly reduces the CE of all models across datasets; (2) LaSCal outperforms the baselines, achieving state-of-the-art results on the datasets and settings we experiment with. We will add a summary of these results in the camera-ready version of the paper (and the full experiments in Appendix).

3. Adding source data to the objective

Incorporating source data could be useful in the scenario where $p_s(X) = p_t(X)$ (please note that this is NOT implied by the label shift definition). We tested this approach empirically by measuring CE on the target distribution of CIFAR-10 using 3 different estimators: (1) Ground truth (with labels); (2) Our estimator (no labels, no source samples); (3) Our estimator + source samples (no labels, with source samples). The results (reported in the Table below) demonstrate that this approach leads to less precise estimates, i.e., yields higher estimated values compared to ground truth (computed with labels).

We also incorporated the estimator including source samples as an objective function for post-hoc calibration (with the temperature scaling method), and we observed no improvement in the results compared to those reported in the paper.