Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Domain Adaptation

Q1: My main concern is that the method is heuristic and not clearly motivated.

A1: While we acknowledge that our method is heuristic, it's important to highlight that the designs of our PseudoCal method are rooted in clear motivations and can be intuitively explained. The inspiration for the PseudoCal framework stems from the factorized negative log-likelihood (NLL) objective. The factorization implies that if two datasets share the same correct-wrong statistics (i.e., an identical count of correct and wrong predictions), they should exhibit similar temperatures after NLL optimization. In the context of PseudoCal, our objective is to synthesize a labeled set that mirrors the correct-wrong statistics of real target samples. We choose to employ mixup for sample synthesis due to its simplicity and its ability to elegantly leverage the target-domain data structure learned by the UDA model. Additionally, leveraging the well-established cluster assumption, we anticipate a sample-level correspondence between mixed samples and real target samples, facilitated by suitable values of the mix ratio $\lambda$. Accurate sample-level correspondences ensure similar correct-wrong statistics, ultimately guaranteeing that the temperature obtained with the labeled pseudo-target set closely approximates the Oracle target-domain temperature.

Q2: It is well known that a mixup training procedure yields more calibrated models. However, it is unclear from the paper why learning Temperature Scaling using a convex combination of pseudo-labeled samples yields meaningful temperature values.

A2: We recognize the effectiveness of mixup training in enhancing model calibration performance. However, in the context of UDA, we observe that this effectiveness falls short of expectations, as illustrated by the calibration results of DINE in Table 6. DINE, a source-free UDA method utilizing training-stage mixup with unlabeled target data, exhibits substantial calibration errors in the trained model. In contrast, our PseudoCal significantly reduces ECE errors for DINE models, as evidenced by the results presented in Table 6.

In explaining the efficacy of PseudoCal, it is crucial to highlight the importance of ensuring similar correct-wrong statistics between the mixed samples and real target samples. This is a key factor because the NLL objective employed by TempScal is directly influenced by both prediction and label information. The factorization of the NLL implies that correct and wrong predictions (evaluated by labels) have contrasting effects on the NLL optimization process. When there are similar correct-wrong statistics, the NLL optimization employed in TempScal becomes alike, leading to similar temperatures. Note that for mixed samples, correctness is assessed based on the predictions of the mixed samples and their corresponding mixed labels. Conversely, for real target samples, correctness is evaluated using the target predictions and unavailable target ground truths. Detailed evidence supporting the similarity in correct-wrong statistics is provided through visualizations in Figure 4 in the Appendix and quantitative analysis results in Table 10 in the Appendix.

Q3: How is this method derived from the UDA situation? The success of a UDA method in different domain shift situations depends on the size of the gap between the domains. I would expect that the UDA calibration would be dependent on how well the UDA method manages to close the domain gap. I don't see how the proposed method handles this aspect of the UDA calibration problem.

A3: One novelty of our PseudoCal approach lies in adopting a target-domain perspective to address the calibration problem in UDA, distinguishing it from existing methods that primarily rely on source data to address the covariable shift challenge. It's essential to note that in the UDA field, various studies, including many in the recently popular source-free UDA settings, also operate without source data. Similar to previous target-specific works, PseudoCal relies on a well-trained UDA model that effectively captures the target-domain data structure. The absence of source data is compensated by leveraging the knowledge embedded in the UDA model. In situations where the UDA model struggles to learn the target-domain structure, the cluster assumption may not hold, potentially damaging the sample-level correspondence between mixed samples and real target samples. This aspect is evident in Table 6, where, in the I$\to$S task, both SHOT and DINE models exhibit low accuracy, resulting in high calibration errors even after being calibrated with PseudoCal. This underscores the importance of a well-trained UDA model in ensuring the effectiveness of PseudoCal.

Q1: Given the first point mentioned in "Strengths", the paper only claimed it as an observation but without further explanation.

A1: The observation stems directly from the inherent characteristics of the factorized NLL objective. As clarified in Section 3.1, the optimization process for correct and wrong predictions in the factorized NLL objective inherently seeks contrasting temperature values. Specifically, correct predictions aim for a small temperature to sharpen the softmax probability distribution, while wrong predictions aim for a large temperature to flatten the softmax probability distribution. Consequently, when dividing the confidence ranges of (0.0, 1.0) into multiple small bins and ensuring two datasets share the same correct-wrong statistics (i.e., an identical count of correct or wrong predictions for each confidence bin), a highly consistent NLL optimization process is expected. This consistency results in obtaining similar temperatures after the optimization process. We hope this clarification provides a better understanding of the rationale behind this observation.

Q2: (a) While the ground-truth labels are not available, can I just assign random labels for each target sample and then adopt PseudoCal (i.e. How is the UDA inference process helpful)?

A2: PseudoCal is a novel and general framework designed to leverage a synthesized labeled pseudo-target set for optimizing TempScal and obtaining an accurate estimation of the Oracle target-domain temperature. The crucial factor ensuring the success of PseudoCal lies in having this pseudo-target set exhibit similar correct-wrong statistics to the real target samples. However, we cannot know the actual correct-wrong statistics of the real target samples due to the unavailability of target-domain ground-truth labels.

Returning to the initial question, while it’s feasible to use randomly assigned labels for calibration with target samples, there is no assurance of favorable calibration performance because the resulting correct-wrong statistics would be entirely random. This is demonstrated in Table 9 under "Pseudo-Label," where we employ hard one-hot target model predictions as labels for real target samples and apply TempScal, resulting in substantial calibration errors.

The effectiveness of PseudoCal with inference-stage mixup relies on establishing a sample-level correspondence between the pseudo-target set and the real target samples, partially guaranteed by the cluster assumption. The UDA inference process is to obtain predictions for mixed samples, and correctness is evaluated based on these predictions and the corresponding mixed labels. This inference process is critical for leveraging the cluster assumption and building the required sample-level correspondence. We have provided comprehensive evidence to illustrate this, including detailed illustrations in Figure 4 in the Appendix and extensive quantitative statistics in Table 10 in the Appendix.

Q3: (b) Since $\lambda$ is a fixed scalar and we do not know whether the target sample prediction result is true or not, what is the point of the last 8 lines in "Analysis" part.

A3: The "Analysis" part in Section 3.2 aims to provide an intuitive analysis and explanation of how mixed samples can exhibit similar correct-wrong statistics as real target data. This is crucial for ensuring that applying TempScal with these mixed samples can yield a temperature similar to the Oracle target temperature. In the revised paper, we have enhanced the presentation for better clarity, incorporating more illustrations of correct-wrong statistics, as demonstrated in Figure 1(b) and Figure 4 in the Appendix.

It's essential to clarify that we analyze the dataset-level similar correct-wrong statistics through the sample-level correspondence of correct or wrong predictions. The definition of correctness differs between mixed samples and real samples. For mixed samples, correctness is assessed based on the predictions of the mixed samples and their corresponding mixed labels. Conversely, for real target samples, correctness is evaluated using the target predictions and unavailable target ground truths. PseudoCal never accesses target ground truths and relies solely on target images and their model predictions to synthesize a labeled pseudo-target set, comprising mixed samples and mixed labels. For a quantitative analysis of the sample-level correspondence, we kindly refer the reviewer to Appendix D.

The use of target ground truths serves exclusively for analysis and explanation purposes to study the success of PseudoCal. This involves explaining and showcasing similar correct-wrong statistics between the pseudo-target set and real target samples, as discussed in both the Analysis part in Section 3.2 and Appendix D.

Q1: The novelty is weak because mix-up is a well-known strategy in image classification tasks. From my perspective, the methodology is the same as the previous ICLR 2018 work, "mixup: BEYOND EMPIRICAL RISK MINIMIZATION".

A1: We appreciate the reviewer's comments and respectfully disagree with the assertion that “the methodology is the same as the previous ICLR 2018 work [1]”.

1. While acknowledging the foundational work of [1] and its impact on the field, we highlight distinct differences in how mixup is employed in our work, as discussed in Section 4.3 "Comparison with training-stage mixup" and summarized below:

• Different application stage: Unlike existing mixup works [1, 2], which leverage mixup during the training stage for model training, our PseudoCal approach employs mixup during the inference stage, specifically for UDA model calibration, without altering model parameters.

• Different mix ratio ($\lambda$): In prior works [1, 2], effective mixup commonly utilizes $\lambda$ values near 1, typically randomly sampled from a beta-distribution, i.e., $\lambda \sim Beta(\alpha, \alpha)$, where $\alpha \in [0.1, 0.4]$. Notably, $\lambda$ values significantly deviating from 1.0 are considered ineffective due to the manifold intrusion problem [2, 3]. On the contrary, PseudoCal exhibits optimal performance with fixed $\lambda$ values significantly deviating from 1.0. However, employing $\lambda=1.0$ in PseudoCal (i.e., PseudoCal($\lambda=1.0$)) leads to significant calibration errors.

• Different calibration performance: We present compelling evidence from two perspectives. First, we examine DINE [4], a source-free domain adaptation method employing the training-stage mixup strategy [1, 2] during model training. As outlined in Table 6, our observations indicate that UDA models trained with DINE still manifest substantial calibration errors. In contrast, our PseudoCal effectively calibrates the DINE model, leading to notable reductions in calibration errors. Specifically, the ECE (%) error drops from 21.81 to 12.82 on DomainNet and from 58.85 to 47.76 on ImageNet. Furthermore, when we substitute our mixup strategy with the training-stage mixup [1, 2] in our PseudoCal framework, denoting this variant as "Mixup-Beta" in Table 9, we observe the superior performance of our mixup strategy. These significant performance differences underscore the distinct nature and advantage of our mixup strategy.

2. Moreover, it is crucial to highlight that our usage of mixup serves as only one technical facet of our framework; the novelty of our paper extends far beyond this component, as summarized below.

• Novel target-domain perspective: We approach the challenging calibration problem in UDA as an unsupervised calibration issue in the target domain, diverging from the previous treatment as a cross-domain covariate shift problem seen in competing approaches listed in Table 1 of our submission.

• Novel TempScal objective factorization: We introduce a unique factorization of the negative log-likelihood (NLL) objective used in TempScal, a distinctive feature not identified in relevant UDA model calibration studies compared in Table 1.

• Novel PseudoCal framework: Our innovative post-hoc framework, PseudoCal, addresses model calibration in domain adaptation through a novel strategy for synthesizing pseudo-target samples. This sets it apart from competing methods, as thoroughly demonstrated in Table 1 of our submission.

• Comprehensive empirical study: Our extensive experiments with PseudoCal encompass 10 domain adaptation methods across 5 UDA scenarios. Notably, we include the calibration under source-free UDA settings and domain-adaptive semantic segmentation for the first time. PseudoCal consistently exhibits significant outperformance over all competing methods.

Q2: The mix-up is conducted at input level, which would introduce artifacts. It would be good to include some visualization in the appendix together with quantitative results.

A2: Thank you for your valuable suggestion. We have added visualizations of input-level mixup for all UDA benchmarks in Figure 5 of the Appendix for a comprehensive understanding.

Q1: The analysis of the mixup in the inference strategy is not sufficient. The comparison between the training stage mixup strategy in other methods and the inference stage is not obvious, the authors should strengthen your highlight.

A1: In Section 4.3, we delve into a detailed analysis of the mixup operation's impact on the calibration performance of our PseudoCal. Two key implementation-level factors are considered: the use of hard one-hot labels or soft predictions for the mixup process in Equation 3, and the value of the mix ratio $\lambda$. Empirical results across various UDA methods and settings are presented in Figure 3(c)-(d). Specifically, we highlight the crucial role of $\lambda$ values, emphasizing in the main text that "a $\lambda$ value closer to 0.5 generates more ambiguous samples, resulting in increased wrong predictions, whereas a $\lambda$ value closer to 1.0 has the opposite effect." For readers interested in detailed quantitative analysis, we provide a comprehensive discussion in Appendix D.

In comparing our approach to mainstream training-stage mixup [1,2], our goal is to highlight the novel aspects of our mixup strategy. We emphasize three key novelties. Firstly, we employ mixup during the inference stage rather than the traditional training stage. Secondly, our mixup strategy exhibits effectiveness with a fixed $\lambda$ value significantly deviating from 1.0, a departure from the prevalent use of values close to 1.0 in existing mixup approaches. Thirdly, applying training-stage mixup directly to train a UDA model, as seen in UDA methods like DINE, still results in substantial calibration errors. Conversely, our PseudoCal can significantly reduce ECE errors for such models, as evidenced by the DINE results presented in Table 6. Additionally, in the ablation study of PseudoCal, where we replace our $\lambda$ usage with values generated through the beta-distribution employed in existing mixup, the results under "Mixup-Beta" in Table 9 show that our mixup strategy outperforms traditional mixup.

Q2: Writing can be improved. In addition, the experiment results could present more in the “Accuracy” metric, which is also intuitive.

A2: Thank you for your valuable feedback on writing. We have revised the manuscript to enhance clarity and provide more detailed results. Regarding the "Accuracy" metric, we have clarified in the first paragraph of Section 4.2 that, in all tables, "‘Accuracy’ (%) denotes the target accuracy of the fixed UDA model." This addition only serves to offer readers a clear understanding of the UDA model's discriminative ability in the target domain. As indicated in Table 1 and Figure 2, our PseudoCal method is a post-hoc calibration approach that solely utilizes the UDA model for inference without changing "Accuracy" of the model. It's worth noting that many existing calibration methods, such as TempScal, CPCS, and TransCal, are also post-hoc in nature.

Q3: How many pseudo-target samples will be generated? Does the number of $n_{pt}$ be the same with target sample number?

A3: Yes, for all experiments, the number of $n_{pt}$ is almost the same as the target sample number $n_t$. Specifically, in the Implementation details, we mentioned that “We use the UDA model for one-epoch inference with mixup to generate the labeled pseudo-target set.” Additinoally, as indicted in our pseudocode in Appendix A, during the inference process, we only mix real target samples that have different pseudo labels, i.e., cross-cluster perturbation. Therefore, in general, $n_{pt}$ would be slightly smaller than $n_t$ due to ignoring the pairs of the shuffled real target samples that have the same pseudo label.

Q4: To find the value of mix ratio \lambda, the authors analyze the ECE on different UDA scenarios. However, it seems that the black-box source-free scenario does not have a corresponding analysis.

A4: We appreciate the thorough examination of our ablation experiments in Table 9 and analysis experiments in Table 10 (Appendix). Both tables maintain a consistent setting for UDA methods. In our initial submission, we presented the black-box source-free method DINE and the white-box source-free method SHOT under the same parent UDA setting, i.e., source-free UDA. Consequently, we only reported SHOT ablation results. Following the valuable suggestion, we have included the results of DINE in both Table 9 and Table 10 to facilitate comprehensive comparisons without altering our initial conclusions.