Improving self-training under distribution shifts via anchored confidence with theoretical guarantees

We sincerely appreciate Reviewer hP9c's insightful suggestions and thoughtful comments. We are pleased that Reviewer hP9c acknowledges our core technical contributions to showing the effectiveness of high-quality uncertainty estimation in the temporal ensemble for improving self-training with solid theoretical and empirical results. Below, we have carefully addressed Reviewer hP9c’s valuable comments, which we believe significantly improve our work. (Unfortunately, due to the number of comments from Reviewer hP9c, our rebuttal addresses only Weaknesses. We will post the remaining responses to Questions in the official comment and apologize for any inconvenience.)

Note: [G#] is the reference for the global responses.

W1: Yes, but we hold a positive view on using the provided default hyperparameter values in diverse distribution shift scenarios. Specifically, we note that the single configurations ($\lambda=0.3$ and $\beta=0.9$) work well for 105 numbers of different distribution shifts with varying difficulties. Also, the performances are shown to be not drastically affected by changes in the hyperparameter values. Finally, we kindly refer to [G4 in the global response] for more detailed explanations of our rigorous hyperparameter selection procedure and its strong practical implications.

W2: We kindly refer to [G1] for the validity of the assumption of $\bar{p}(x; c_{0:m})$ made in Theorem 3.1. Also, we note that AnCon’s strong empirical performance can be preserved even if the condition is violated as discussed in [G3].

W3-1: Thank you for pointing out the important missing details in the error bars. In the original version, we opted to omit the variance terms because the variance is not particularly large for any method (e.g., on average 0.96 for OfficeHome), which gives statistical significance at a P-value < 5%. However, in order to prevent concerns about the statistical significance, we have added P-values for each table.

W3-2: Thanks for the great suggestion that can help readers to understand the uncertainty estimation behavior of AnCon. Following the Reviewer hP9c’s insightful suggestion, we have analyzed the reliability diagram. As widely known in the literature, all self-training methods turn out to be overconfident. However, as shown in Figure R5, AnCon relaxes the overconfident behavior, which can be inferred from its better ECEs in the manuscript (e.g., Table 5).

W3-3: To clarify, AnCon aims to solve the source-free domain adaptation (SFDA) and test-time adaptation (TTA) problems, not the unsupervised domain adaptation (UDA) problem wherein the labeled source domain dataset is available during the adaptation. The unavailability of any reliable labeled samples in SFDA and TTA makes more challenging learning scenarios, which involve early learning phenomenon and model collapse under severe distribution shifts (cf. Figure 1 (a)). Therefore, the UDA methods would not be directly comparable to AnCon. Also, we note that the chosen baselines show state-of-the-art performances in general settings (cf. [G5 in the global response]).

W4: Kindly see [G3] for the unique and significant performance gains from AnCon in challenging distribution shift scenarios, wherein the average correct predictions assumption is violated.

W5-1: As Reviewer hP9c commented, regularity conditions in Assumptions A.1, A.2, and A.3 are mild but essential for most theoretical studies with convergence analyses. Assumptions A.1 and A.2 can trivially hold under bounded parameter values that can be guaranteed by optimizing neural networks with finite iterations under a gradient or weight clipping. Assumption A.3 holds for infinite-width neural networks, i.e., the neural tangent kernel (NTK) regime (Charles & Papailiopoulos, 2018). Given that the gradient descent training dynamics of neural networks can be well approximated by NTK (Jacot et al., 2018), the PL condition can be generally regarded as a mild assumption. We fully agree with Reviewer hP9c’s suggestion and have added these discussions to “Appendix A.1 Assumptions.”

Charles, Z., & Papailiopoulos, D. (2018). Stability and generalization of learning algorithms that converge to global optima. In ICML. Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. In NerIPS.

W5-2: Thanks for the valuable clarifying questions! $f_k(x; \theta)$ is right; $\bar{f}(x) := (\bar{f}_1(x), \bar{f}_2(x), \cdots, \bar{f}_K(x))$; $\hat{E}$ is the Monte-Carlo estimator with mini-batch samples; we have introduced $\sigma^{2}(\bar{\theta})$ for the variance of stochastic gradient of $\bar{f}$; finally, we introduce a new notation for the softmax as $\phi$ instead of $\sigma$ to avoid the confusion.

W5-3/4: Thanks for the great suggestions. We have added links for all acronyms and increased the font size for all small figures.

Q1-1: We inadvertently caused confusion to Reviewer hP9c on the connection between AnCon and knowledge distillation (KD). To be clear, we do not intend to treat the pseudo label $\hat{Y}$ as the true label $Y$ since this reduces the theoretical rigor of our work. Indeed, we clearly state the bias coming from pseudo labels in the optimality gap in Theorem 3.2. Also, we state that “the gradient is biased due to the usage of pseudo labels and the generalized temporal ensemble” in Line 168-169. The purpose of this connection is to show how AnCon can reduce the variance of stochastic gradients in the self-training scenario through the partial variance reduction theory by interpreting our generalized temporal ensemble as the teacher network. In order to avoid any confusion, we have changed Lines 165-169 as follows:

“”” … , where the generalized temporal ensemble is the teacher network $f^{(t)}$ with a notable difference that the pseudo label $\hat{Y}$ is used instead of the true label $Y$ which requires careful analysis for studying the optimality. The purpose of this connection is to perform a convergence analysis of AnCon by modifying the partial variance reduction theory [31], as given below. We note that the usage of pseudo labels, instead of the true label, and the generalized temporal ensemble result in an inherently biased gradient estimator. Therefore, the convergence analysis requires special treatments for handling the biased gradient, unlike typical supervised learning settings in self-distillation literature.
“””

Q1-2: It can be derived by using the law of total expectation with the random event $1(Y(X) = \hat{Y}(X))$ and then applying the bounded support assumption, which we have added to Line 678.

Q2-1: If this refers to Line 125 in Eq 3, yes due to the recursion.

Q2-2: It is \sum_{0}^{m}, which means the current average confidence is taken into account because the exponential moving average is introduced to avoid the computation of the expectation with respect to all samples.

Q2-3: Thank you so much for your careful review and pointing out the type. It is $\theta_m$ because $Err_{XY}(\hat{Y}; \theta_{m,0})$ is an error rate of pseudo label under parameter $\theta_m$.

Q2-4: Thanks again for pointing out the typo. This is $w_{0:m}$ because the generalized temporal ensemble includes the initial model prediction. (After very careful review, we have found other typos in Line 178, Line 179, Line 674, which we corrected).

Q3: AnCon has almost the same computational complexity as ELR, and the confusion of Reviewer hP9c would be due to our undefined notation of $\hat{E}[c(X; \theta_i)]$ which is a Monte-Carlo approximation of $E[c(X; \theta_i)]$ with mini-batch samples $\xi$. Specifically, Eq (2) for storing previous predictions is the same as with storing the previous logit vector in ELR ($K \times N$ numbers). Also, Eq (3) can be efficiently implemented by using softmax output values during the training process which are already computed for computing the self-training loss. Thus, Eq. (3) requires storing only a single number with the computational cost of adding $B$ numbers. In order to prevent the confusion, we have added the above discussion in Line 131.

Q4: Kindly see the response to the weakness.

Q5: As per Reviewer’s insightful comment, we have tested the undiscounted vanilla temporal ensemble (UVTE) method, which results in 4.2% of reduction of accuracy on average. Intuitively, UVTE does not work well because memorizing all predictions, which involve highly inaccurate ones, can result in a low-quality temporal ensemble. Indeed, this is well illustrated in Figure R3 where the UVTE would not take advantage of gathering more samples due to inclusion of more incorrect predictions compared to AnCon. We believe this new result further highlights the importance of uncertainty awareness for the temporal ensemble in self-training, which is consistent with our theoretical results in Theorems 3.1 and 3.2.

Q6: Thank you for the insightful question that made us think about the non-trivial extension of AnCon. Extending AnCon to sequential decision-making scenarios presents significant challenges as they involve the fundamentally different mechanisms. In sequential decision-making, leveraging observed rewards and balancing exploitation and exploration are core aspects (e.g., constructing the upper confidence bound of the reward in the bandit problem), unlike in self-training. Therefore, the main challenges for applying AnCon to this setting would be defining rewards and incorporating exploration strategies into pseudo label generation. As it holds great significance, we mark this extension as an important future direction.

“”” Assume $l(\theta)$ satisfies $L$-smoothness, $\mathcal{L}$-expected smoothness, and $\mu$-Polyak-Lojasiewicz (PL) condition (cf. Assumptions A.1, A.2, and A.3). For $\gamma \leq \tfrac{\mu}{4 \mathcal{L} \cdot L}$ and a carefully chosen $\lambda$ (cf. Lemma A.6), it holds that $Eq. (12)$

Further, if $\mathbf{w}_{0:m}$ such that {...}, i.e., the teacher has a sufficiently good performance, the optimality gap becomes

$Eq. (7)$

where $l^*$ {...} .

“”” (here we could only refer equation number and use {...} to represent the repetitive equations since the MathJax cannot display the equations properly.)

Modified discussion in Lines 187-190: “”” In Theorem 3.2, we remark that the vanilla self-training ($\lambda=0$) results in the maximum value of the last term in Eq. (12), which is the neighborhood size of the stochastic gradient descent, while having the same first two terms. In other words, the result suggests that AnCon is at least better than vanilla self-training under the mild regularity conditions. Furthermore, under the assumption of a sufficiently good performance temporal ensemble, we gain valuable insights on the design of $w_{0:m}$: aiming for … “””

We sincerely thank Reviewer qtUp for the insightful suggestions and constructive comments, which have significantly improved the clarity of our paper. We are glad that the reviewer enjoyed reading our paper and acknowledged our principled improvements over the existing temporal consistency methods with theoretical guarantees and attractive properties. In our response below, we have addressed the questions and comments. Note: [G#] is the reference for the global responses.

W1-1: To clarify, AnCon aims to solve the source-free domain adaptation (SFDA) and test-time adaptation (TTA) problems, not the unsupervised domain adaptation (UDA) problem wherein the labeled source domain dataset is available during the adaptation. The unavailability of any reliable labeled samples in SFDA and TTA makes more challenging learning scenarios, which involve early learning phenomenon and model collapse under severe distribution shifts (cf. Figure 1 (a)). Therefore, the UDA methods would not be directly comparable to AnCon. Also, we note that the chosen baselines show state-of-the-art performances in general settings (cf. [G5 in the global response]).

W1-2: Our suboptimal choices of hyperparameters are due to our rigorous and practical hyperparameter choice setting, and we kindly refer to [G4] for detailed explanation on the hyperparameter selection procedure and its strong practical implication.

W2: Thanks for pointing out the significance of the assumption made in Theorem 3.2, which asserts that "AnCon is at least better than vanilla self-training." However, the assumption is required only for deriving the final bound in Eq (7) to gain more insights about the impact of the generalized temporal ensemble's quality on the suboptimality of self-training. Rather, the core message of Theorem 3.2 that “AnCon is at least better performance than the vanilla self-training” holds without this assumption. Specifically, in Eq. (13) on page 15, $N(\lambda^\dagger) / N(0)$ corresponds to the ratio between the last term of Eq. (12) under AnCon and the vanilla self-training. Crucially, this ratio is less than or equal to 1, supporting the central message without the assumption.

Thanks to the Reviewer qtUp’s insightful question, we have separated the statement of Theorem 3.2 into two parts and modified the discussion in Lines 187-190 (Kindly see the comment below for the final form) to avoid the initial confusion and emphasize that AnCon guarantees at least better performance than the vanilla self-training can hold without such assumption.

W3: Thanks for pointing out the notation mistake. Since the argument of the function $\xi$ is an arbitrary scalar, we have changed $x$ to $z$ in Line 149. We have changed the index notation of $m$ in the Lemma A.4. to $o$. These notational changes do not invalidate the proof. Regarding the confusion from the index notation, kindly see the response to Q3.

W4: Thanks for pointing out the minor typos! We have changed $k$ by $m \cdot q$ and fixed typos for the references.

W5: Kindly see the responses to Q3 and Q4.

Q1: Thanks for suggesting to unify the notation of the generalized temporal ensemble $\bar{f}$ and the pseudo label $\hat{Y}$ that might be clearer than the current notation. Despite careful thoughts, unfortunately, we do not see any feasible way to unify the notation without sacrificing the insights and clarity that our current notation can provide. Specifically, our separate notation enables clear explanations of the selective temporal consistency (cf. Eq (2)) that is only involved in $\bar{f}$. Also, the notation can intuitively explain the role of $\bar{f}$ that regularizes the pseudo label $\hat{Y}$ through label smoothing, which makes the discussions in Lines 105-115 and the connection between AnCon and knowledge distillation clearer. Finally, there are points that require discussion solely on either of them (e.g., when referring to the accuracy of pseudo labels in Line 145 and Line 184; when discussing the quality of the generalized temporal ensemble in Theorem 3.1). Given the above reasons, we hope that Reviewer qtUp could agree with the advantage of separating notation for the pseudo label and the generalized temporal ensemble.

Q2: Thanks for highlighting the validity of the assumption in Theorem 3.1. Here, we emphasize the important aspects of the assumption: the assumption is based on the “conditional” independence between temporal correctness given that the current prediction on a sample is relatively confident. This is grounded in the strong correlation between accuracy and confidence across a wide range of scenarios. Another way to understand this assumption is that the previous history of predictions is summarized in the random event whether the prediction is confident or not. We further remark that the validity of this assumption could be reinforced by strong empirical support for Eq. (6), as shown in Figure R3.

Q3: Thanks for pointing out the unclear notation for the supporting lemma. The number of random Bernoulli samples $m$ in the Lemma A.4 (which we change to $o$) is not used in the other parts. Therefore, all other parts remain the same, except Line 624 which refers to this lemma is changed to “... due to Lemma A.4 with $p_i = P(Y(x) = \hat{Y}(x; \theta_i))$, $o = Q(x; c_{0:m})$, and $q= 1/2$.”

Q4: We kindly refer to [G2] for the clarification of the notion of asymptotic convergence. Also, we note that [G2] presents the important implication of Eq. (6) in the non-asymptotic region, i.e., monotonic improvement of quality of the generalized temporal ensemble, with strong empirical evidence.

Q5: As explained in [G2], we kindly refer to Figure R3 that shows monotonic improvements over 100 epochs of quality of the generalized temporal ensemble under diverse distribution shifts scenarios, being consistent with Theorem 3.1.

Thanks for reading the authors' rebuttal and asking a further clarifying question. 

The confusion of Reviewer qtUp is noted. In our changed notation, all $m$ in Lemma A.4. are changed to $o$, which is the number of random Bernoulli samples. We remark that Lemma A.4 is a supporting lemma, which means that the semantics of notations are independent of other parts of the paper (i.e., none of the quantities in Lemma A.4, including Eq. (30), is interpreted as the number of models). 

Therefore, all $m$ notations in Lemma A.4 are changed to $o$. Then, the right-hand side of Eq. (31) is $o q - o \bar{p} - o q \log \frac{q}{\bar{p}}$, which is obtained by $M(t) = exp( o \bar{p} (exp(t) - 1))$ and $t = log \frac{q}{\bar{p}}$. 

From the above, Eq. (29) is $p(S_o \leq q o) \leq exp(o (q - \bar{p} - q \log \frac{q}{\bar{p}}) )$. Therefore, applying this inequality to Theorem 3.1 with $p_i  = p(Y(x) = \hat{Y}(x; \theta_i))$, $o = Q(x; c_{0:m})$, and $q = 1 / 2$ proves the last inequality in Eq. (11). 

We hope this clarifies why there is no mistake in the proof and that the confusion arose from the same notation used in the supporting lemma and other parts of the paper. We sincerely appreciate Reviewer qtUp's thorough review and attention to detail.

Q5: We kindly refer to “Section 4.1” for the integration of AnCon with state-of-the-art SFDA technique (NRC) and TTA technique (GCE). We also kindly refer to [G5 in the global response; GLOBAL SOTA] for conceptualizing the state-of-the-art performance levels of considered baselines. Regarding potential benefits, AnCon differs from state-of-the-art methods in handling noisy pseudo labels under distribution shifts. Specifically, NRC filters incorrect predictions based on local consistency, while AnCon uses temporal consistency. Combining NRC and AnCon leverages pseudo labels that are both locally and temporally consistent, resulting in significant performance improvements over NRC or Self-Training + AnCon (cf. Table 1). In addition, GCE reduces the impact of wrong pseudo labels rather than finding them. Applying GCE to AnCon minimizes the effects of potentially wrong but temporally consistent pseudo labels, which can be implied by the performance of GCE + AnCon compared to GCE or Self-Training + AnCon (see Table 1). In summary, AnCon can complement existing state-of-the-art methods by handling noisy pseudo labels fundamentally differently.

Q6: We kindly remind Reviewer fUqB that the concept shift is out of the scope of our work as stated in Line 71. Also, we note that addressing concept shifts in the self-training scenario without labels is questionable, as even detecting such shifts without feedback on predictions through shifted labels in a principled manner is impossible (e.g., Lu et al., 2018).

• Lu et al. (2018). Learning under concept drift: A review. IEEE T-KDE.

Q7: We appreciate the curiosity of Reviewer fUqB about extending AnCon to various neural network architectures and tasks. Indeed, our theory and algorithm for AnCon are broadly applicable as AnCon is independent of specific neural network architectures or dataset structures, unlike methods that rely on batch normalization statistics or random image augmentations. At the same time, being agnostic to architectures and data structures make it inherently hard to make rigorous theoretical statements on the extensibility. Therefore, affirmative conclusions about the extension of AnCon would require large-scale comprehensive experiments with careful design of explorative strategies, which is beyond the scope of this paper. Moreover, even if AnCon performs suboptimally in certain architectures or tasks, this does not invalidate (1) our theoretical contribution of showing how uncertainty-aware temporal consistency improves self-training under distribution shifts and (2) our algorithmic contribution of designing an effective temporal ensemble mechanism without necessitating computationally heavy processes such as additional forward passes or neighborhood searches. In summary, while we acknowledge the potential benefits of further explorations, we believe our current focus provides a solid foundation for understanding AnCon's impact and effectiveness.

We sincerely thank Reviewer fUqB for the valuable suggestions and comments. We are glad that the reviewer acknowledges our rigorous theoretical analyses, efficient algorithm development, and effective performances of AnCon under different distribution shifts scenarios. We believe that addressing the valuable comments further improves our paper, which we hope can effectively address any concerns of Reviewer fUqB.

• [G#] is the reference for the global responses. (Unfortunately, due to the number of comments from Reviewer fUqB, our rebuttal addresses parts of responses. We will post the remaining responses in the official comment and apologize for any inconvenience.)

W1: Thank you for highlighting the novelty aspect of our paper. As Reviewer fUqB noted, promoting temporal consistency is a well-established framework in self-training algorithms. However, our contribution lies in demonstrating that "uncertainty-awareness" enhances the effectiveness of temporal consistency under distribution shifts. Specifically, Theorem 3.1 shows that uncertainty awareness improves the quality of temporal ensembles proportional to the number of temporally confident predictions. Additionally, Theorem 3.2 and Corollary 3.2.1 illustrate that "AnCon" reduces the optimality gap of self-training, which can be further improved by a high-quality, uncertainty-aware teacher. Crucially, such high-quality teachers are driven by our simple thresholding rule (Eq. (3)) with theoretical guarantees (Theorem 3.1). The significant theoretical contribution has not been studied and would not be possible by just incorporating the temporal consistency through ensemble as shown in Figure R6. In order to help readers to conceptualize the core theoretical contributions of the paper, we have the following discussion to Line 199.

‘’’ Furthermore, we highlight the significance of AnCon’s uncertainty awareness for improving effectiveness of temporal consistency in self-training. Specifically, from the result of Theorem 3.1, $g^{KL}(w_{0:m})$, thereby the optimality gap of the self-training, can be further reduced by collecting more confident samples, while guaranteeing the condition $\bar{p}(c_{0:m}) \geq 0.5$. Therefore, it is important to collect high-quality predictions for matching the condition, which would be hard to satisfy under other temporal ensemble mechanisms (cf. Figure R3). ‘’’

W2: Kindly see [G3 in the global reference] that explains the unique and significant performance gains from AnCon in challenging distribution shifts scenarios.

W3: Thank you for pointing out concerns regarding the dependency on the initial model parameter. As Reviewer fUqB conjectured, we have rigorously shown the impact of initial model performance on the sub-optimality of AnCon and the vanilla self-training method (cf. $l(\theta_0) - l^*$ and $Err(\hat{Y}; \theta_0)$ in Corollary 3.2.1). Crucially, in Corollary 3.2.1, both AnCon and the self-training method can improve by reducing the initial optimality gap $l(\theta_0) - l^*$ as the number of inner and outer iterations $T \cdot (m+1)$ increases, which is why we need “adaptation” if the performance deteriorates under severe distribution shifts. Finally, we kindly refer to [G3] for the discussion of empirical results supporting this argument.

Q1: We gently remind Reviewer fUqB that the asymptotic convergence on inputs on which $f$ produces infinitely many relatively confident predictions is rigorously proven in Theorem 3.1 with explicitly stated assumptions. We also kindly refer to [G1] for the discussion about validity of assumptions and [G2] for the empirical evidence.

Q2: This is actually rigorously proven in Theorem 3.2, a core technical contribution of our work, as discussed in Lines 187-199. Specifically, “the underlying mechanism of how label smoothing achieves this reduction” is through connecting AnCon with the self-distillation. Crucially, this connection enables the adoption of the partial variance reduction theory, which can show that the generalized temporal ensemble reduces the neighborhood size of SGD. Our convergence analysis provides the optimality gap in Eq. (7), with the maximum gap occurring in self-training without label smoothing (cf. Line 187-188). This demonstrates that AnCon is at least better than vanilla self-training (cf. Line 189). Additionally, AnCon further improves the optimality gap by reducing $g^{KL}(w_{0:m})$, which explains the effectiveness of selective temporal consistency.

Q3: The impacts of severe distribution shifts typically result in a poorly performing initial model, which can significantly reduce the effectiveness of self-training methods like AnCon. However, as discussed in our response to W3, AnCon successfully improves poorly performing initial model parameters under severe distribution shifts, unlike other self-training methods. Further, following Reviewer fUqB’s suggestion, we performed additional experiments in the highly class imbalance scenario. This setting is significant as effective methods like oversampling are not applicable in self-training, yet they largely impact self-training performance. In the new experiments, we compare AnCon with ELR and vanilla self-training under a heavy-tailed class imbalance with an imbalance ratio of up to 65 times. As shown in Figure R2, AnCon's effectiveness is well-preserved under severe class imbalance scenarios. In summary, we believe AnCon would show promising performances in multiple challenging self-training scenarios.

Q4: Kindly see [G4] for the detailed explanation of our rigorous hyperparameter selection procedure and its significant practical implications. We also kindly refer to “Section 4.3.2. Robustness to the choice of hyperparameters” which shows the stable performances of AnCon even under arbitrary choices of hyperparameters.