Source-Free Target Domain Confidence Calibration

Thank you for your comments and feedback.

Q1. The paper is hard to read. The author should split the chapters to make them easier to read, for example, the third section should be split appropriately.

A. In the revised version we'll rearrange the paper structure to improve readability.

Q2. I am a little confused about this setting, i.e., calibration via the unlabelled data from the target domain. In what scenarios would this setting be used? TransCal realizes calibration via the labeled source domain data. The author may discuss the differences and applications between the two settings.

A. The distinction between SFCC and TransCal lies in the fact that TransCal utilizes labeled source data for model calibration. In contrast, our approach tackles a more challenging calibration scenario where source domain data is entirely inaccessible, often due to privacy concerns. As a result, TransCal cannot be applied in this context.

Q3. The pseudo label has been explored thoroughly in semi-supervised learning. Simply transferring it to the calibration lacks novelty.

A. As Fig 4a shows, PL are usually very noisy (20%) and semi-supervised methods cannot simply consider them as true labels and they need to filter out samples usually based on the prediction confidence. The main observation of our study is that, in spite of the high noise level of EPLs, we can directly use them for calibration because estimating the network accuracy based on the pseudo label is very similar to the true accuracy (as shown in Fig 4b). See also our joint answer to all the reviewers.

Q4. The author optimizes the temperature via the adaECE loss. For a fair comparison, the author should report other evaluation metrics such as SCE.

A. Thank you for the feedback. In the updated version, we include additional evaluation metrics such as NLL, Brier score and SCE (see A.2). The results show that SFCC outperforms other calibration methods on the new evaluation metrics.

Q5. line 309. "we followed the evaluation protocol described in the TransCal Paper, which involves splitting each target domain into 80% for training and 20% for validation". It seems that TransCal split the training set and validation set on the source domain.

A. Thank you for pointing this out; it is indeed a misleading explanation, and we will revise it in the updated version. What we intended to convey is that we applied the same data split on the target domain as was done on the source data in the TransCal paper.

As suggested, in A.1 we added a scheme that described the components and the flow of our method.

Thank you for your comments and feedback.

Q1. Why do you think the noisy label would have the same effect as the correct label in calibration? Is there any empirical observation or theoretical guarantee? I doubt this motivation.

A. Fig 4b exactly shows that for all the source-target pairs we checked and for all confidence bins we get that the estimated accuracy based on the pseudo-labels is very close to the true accuracy (see also joint answer to all reviewers).

Q2. In Eq. 4, what is the difference between the Pseudo label and the predicted label?

A. A pseudo label is obtained by applying the source model to a sample from the target domain. The Enhanced Pseudo Label is generated by combining a robust pre-trained feature extractor with the pseudo label. The predicted label represents the model's prediction after adapting it to the target domain.

Q3. Using $\hat{A}$ to $A$ is not suitable as they would exist a big gap (you do not even know how reliable the pseudo labels are). $\hat{A}{i}$ would not be equal to $A{i}$ in this case. Also, the approach from the left to right in Eq.6 should not be held.

A. Indeed even enhanced pseudo-labels can be very noisy (see Fig 4a). Our main novel observation is that the EPL label noise follows a specific pattern such that, despite the high noise level, estimating the bin-wise accuracy of the adapted model on the target domain using either the true labels or the pseudo-labels yield very similar results. Figure 4b provides empirical evidence for this observation.

Q4. The reason in Fig. 4b may be the fact that you are using a strong pre-trained network, which should not be concluded as a common empirical observation.

A. We do use a strong pre-trained network to extract more accurate pseudo labels. However, as Fig 4a shows, these pseudo labels are still very noisy. The main observation of this study, shown in Fig 4b is that in spite of the high noise level of EPLs, we can directly use them in the calibration process.

Q5. Are the bins divided manually with a fixed number?

A. Following previous works we used a fixed number of bins (15). The AdaECE bins' boundaries are set to have the same number of points in each bin.

Thank you for your comments and feedback. Q1. The technical novelty is not clear. A. See joint answer to all reviewers.

Q2. The presentation is hard to understand, especially the methodology introduced in Lines 191-208. It is confusing and weak to claim that two versions of $A_{i,1}$ are equal by definition. By which definition? Presentation from Line198 to Line215 is only based on assumptions without any theoretical or generalized empirical guarantee as the support.

A. The definitions of the two versions of $A_{i, 1}$ are identical, since they both count the amount of points for which $y_t=\hat{y}_t=\tilde{y}_t$. The claims in line 198-215 are empirically justified across hundreds of source-target pairs in Fig 4b.

Q3.1: only three SFDA methods are not enough. Since the claim of this submission is a calibration method for SFDA, only if the authors believe that the selected three SFDA methods (SHOT, AaD, and DCPL) can fully represent existing SFDA approaches, however, is it a fact?

A: We selected three SFDA methods: SHOT, AaD and DCPL which represent three different approaches in regard to our generated pseudo-labels that are relevant to be tested with our calibration method. One method, DCPL, uses the same pseudo-labels (EPL) without changing them during training. A second SFDA method, SHOT, which uses pseudo-labels that rely only on the source model and not on a strong pre-trained network (and also adapted during training). A third method, AaD, which does not use pseudo labels at all in the target adaptation training.

Q3.2: It is required to do ablations on GEPL with other techniques of improving pseudo labels such as the common thresholding-based method. Third, other calibration error metrics mentioned in TransCal [3] excluding ECE should be reported because ECE could be misleading sometimes.

A: Thank you for your feedback. In the revised version of the article appendix , we've included additional evaluation metrics (In A.2), such as NLL and the Brier score, which were used in the TransCal paper. Regarding your question about improving pseudo labels using other techniques, we experimented with different feature extractors and present the results from the best-performing one. If we understand correctly, common thresholding-based methods remove examples we're uncertain about. We chose not to pursue this approach because we believe that excluding uncertain examples would be detrimental to the calibration process, as they are essential for it.

Q3.3: In addition, what is the accuracy of all SFDA models?

A. In section A.3 we have added tables that present the accuracy of all SFDA models.

Thank you for your comments and feedback.

Regarding the novelty, please refer to our response to all reviewers.

We have included in A.5 a comparison with PseudoCal on the VisDA dataand showed that our method works better. In the final version we will include PseudoCal results for all the tasks,

Q1. Will the proposed EPL method also benefit the SFDA performance, compared to the baseline methods, SHOT and DCPL?

A. The EPL method can indeed contributes to SFDA performance. EPL was introduced in co-learning paper (Wenyu et al, 2023) and was also applied in DCPL, extracting better pseudo-labels. Both co-learning and DCPL showed the contribution of EPL to performance compared to SFDA baselines such as SHOT and AaD.

Q2. For calibration, how was the bin number decided? Would variations in bin number affect the calibration outcomes?

A. We selected the number of bins based on those introduced in the compared methods: CPCS, TransCal, and UTDC. Moreover, we experimented with different bin counts and included an example of the results A.4.

Q3. Are Figs 1 and 2, show results recorded at the end of the adaptation process?

A. Yes, in all analysis or results which are related to the adapted model, we used the model obtained at the end of the adaptation process.

Q4. For temperature scaling, is the optimal temperature calculated on the entire dataset or per mini-batch?

A. The optimal temperature was calculated on the entire validation set of the target domain by minimizing the adaECE or ECE.

Q5. Clarifications Needed:

Q6.1: In lines 504-510, the argument about classifying difficulty lacks clarity—how does this paragraph validate that claim?

A. We aim to demonstrate that examples with incorrect pseudo-labels are inherently difficult to classify. To quantify the difficulty of an example, we propose measuring the difference between the distance to the nearest center and the distance to the second-nearest center. Intuitively, easily classifiable examples tend to have one dominant prediction, meaning the nearest center is significantly closer than all others. We confirmed that, in cases where pseudo-labels were incorrect, there were typically at least two comparably plausible predictions, unlike examples with correct pseudo-labels.

Q6.2: Does Figure 4b account for outliers, and could they impact the interpretability of the results?

A: We are not sure we fully understand the question. In Fig 4b. there were several outlier source-target pairs in which the accuracy estimation was relatively low, but in the vast majority of cases, the accuracy estimation using EPL was very high.

Q6.3: Figure 3b appears to support Equation (6), but the connection of Eq. (6) to Figure 4b is unclear for me. Please correct me if I've misunderstood anything.

A. Figure 4b shows that although EPL is very noise (Fig 4a), the difference between computing adapted network accuracy using the correct labels and estimating the accuracy using EPL is very small. This is the main empirical justification of our method presented in Eq. 6.