Adaptive Energy Alignment for Accelerating Test-Time Adaptation

Further analysis and discussion on AEA's TTA acceleration

Thank you for your valuable suggestions. Our AEA facilitates model adaptation by aggressively reducing the energy gap from the early online batches, which allows rapid alignment of the target feature with that of the source domain. Notably, previous studies have already suggested that the feature alignment between the source and target domains can alleviate the domain gap and achieve better adaptation. Consequently, as we discussed in Sec. 4.3, our AEA turns out to promote fast feature alignment, leading to improved performance in the early adaptation batches (i.e., acceleration). We have discussed these points in the subsection titled “Why does energy alignment lead to better adaptation?” in Sect. 4.3, and endeavored to describe them as thoroughly as possible in Fig. 7.

 

To further demonstrate that our energy reduction method indeed aligns the true source and target features rapidly, in Table 1, we measured the true distance between source features and target features during TTA adaptation. Specifically, for each adaptation stage, we (1) estimate the source/target feature means for each ground-truth class, (2) measure the Euclidean distance between the feature means for each class, and (3) take the average across all classes. These measurements can indicate how well the adapted model performs feature alignment between the source and target domains. As can be seen, our AEA achieves better alignment by reducing the distance between each feature from 5.18 to 0.98 over 40 adaptation batches, whereas TENT does from 5.18 to 1.76. Based on these findings, we argue that our adaptive energy alignment approach facilitates more effective feature alignment from the early online batches, accelerating TTA adaptation. We have supplemented this additional analysis in Sec. A.8 of the Appendix, and we appreciate your comments.

 

• Table 1. Feature alignment between the source and target domains. We measure the true distance (i.e, Euclidean distance) between source features and target features during TTA adaptation. Each column represents the number of adaptation batches. Experiments are conducted on CIFAR10-C (gaussian noise with severity level 5).

	0	10	20	30	40	50	60	70	80	90	100	110	120	130	140	150
No adaptation	5.18	5.18	5.18	5.18	5.18	5.18	5.18	5.18	5.18	5.18	5.18	5.18	5.18	5.18	5.18	5.18
TENT	5.18	1.87	1.83	1.80	1.76	1.73	1.69	1.64	1.61	1.58	1.55	1.52	1.49	1.46	1.44	1.41
AEA (ours)	5.18	1.58	1.28	1.09	0.98	0.94	0.89	0.86	0.82	0.79	0.78	0.77	0.75	0.71	0.69	0.71

Relevant details in the main table captions

Thank you for your comments. To ensure a fair comparison, we have reproduced the experimental results of each method following the setup of a recent benchmark [1], which has standardized the TTA setup with consistent configurations (e.g., pre-trained model, learning rate, batch size, etc.). In accordance with the reviewer’s suggestion, we have included the relevant details in the table captions of the revised manuscript.

 

[1] Zhao et al. On pitfalls of test-time adaptation. ICML’23.

We appreciate the reviewer for the constructive feedback, and taking a positive position. We tried to carefully address the raised points. If there are any remaining concerns, we would be grateful for another chance to respond.

Evaluation on the other source-free scenarios (e.g., SFDA)

Thank you for your interesting suggestion. Our AEA can also be applied to the SFDA task. During the rebuttal period, we conducted additional experiments and present the results in Table 1. The main difference between SFDA and TTA is that SFDA allows access to the entire target samples for model adaptation, whereas TTA should conduct adaptation over a continuous stream of target batches. To demonstrate the effectiveness of our AEA, we measured performance by varying the number of target samples for adaptation in the SFDA setting (but, the performance is evaluated over the entire target samples). The results show that our method still achieves competitive performance in the SFDA setting, with a significant performance improvement when the number of target samples is small. For example, given only 30% of the entire target samples (for adaptation), our AEA outperforms SHOT by 6.9% and even surpasses the performance of SHOT adapted on the full dataset. These findings further support the advantages of our method in accelerating adaptation and suggest its applicability in other source-free scenarios.

 

• Table 1. Performance under SFDA setting. We report classification accuracy (%, ↑) with varying the sample sizes (in column) for model adaptation. Experiments are conducted on CIFAR10-C (gaussian noise with severity level 5). Other experimental details are provided in Sec. B.4 of Appendix.

	10%	20%	30%	40%	50%	60%	70%	80%	90%	100%
No adaptation	26.9	26.9	26.9	26.9	26.9	26.9	26.9	26.9	26.9	26.9
TENT	61.6	62.9	64.3	64.9	66.6	67.5	68.7	69.1	70.1	70.5
SHOT	63.2	67.8	70.2	71.3	72.6	73.3	74.3	74.5	75.5	75.2
AEA (ours)	67.7	72.9	77.1	77.4	77.6	77.6	77.8	77.7	77.2	76.8

 

To further confirm the versatility of the proposed method, in Table 2, we also include results in a continual TTA setting where the target domains continuously change. In this scenario, we adopted a simple model restoration approach from [1] to prevent error accumulation. We observed that AEA achieves a satisfactory performance even in this challenging scenario, further demonstrating the strong applicability of our approach.

 

• Table 2. Classification errors (%, ↓) under continual TTA setting on CIFAR100-C. Experimental details are provided in Sec. B.9 of Appendix.

	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elastic	Pixel	JPEG	Avg.
No adaptation	73.0	68.0	39.4	29.4	54.1	30.8	28.8	39.5	45.8	50.3	29.5	55.1	37.2	74.7	41.2	46.5
BN adaptation	42.3	40.8	43.2	27.7	41.8	29.8	27.9	35.0	34.8	41.7	26.4	30.2	35.6	33.2	41.2	35.4
TENT	37.2	34.3	36.3	27.3	39.4	30.1	27.1	32.4	33.1	35.8	24.8	28.0	35.2	30.8	39.1	32.7
EATA	37.2	33.1	36.2	27.8	37.9	29.7	26.9	32.7	31.3	35.3	26.9	28.7	33.4	29.8	37.4	32.3
SAR	40.5	34.9	37.1	25.9	37.2	28.0	25.6	31.8	30.8	35.9	25.2	28.1	32.1	29.2	37.3	32.0
ROTTA	49.1	45.0	45.5	30.2	42.6	29.5	26.0	32.3	30.6	37.6	24.7	29.2	32.7	30.4	36.9	34.8
COTTA	40.6	38.0	39.9	27.2	38.2	28.5	26.5	33.3	32.2	40.6	25.1	26.9	32.3	28.3	33.8	32.8
AEA (ours)	36.2	32.8	32.8	25.3	36.3	27.5	24.8	29.7	29.0	31.8	24.0	25.3	32.2	28.1	35.8	30.1

 

Thanks again for your suggestion and we have included these additional results and details in the Appendix of our revised manuscript (SFDA in Sec. B.4 and continual setting in Sec. B.9).

 

[1] Wang et al. Continual test-time domain adaptation. CVPR'22.

Dear Reviewer SzkH,

We would like to express our gratitude once again for your constructive comments and positive evaluation of our work. As the discussion period is ending, we respectfully remind you of our response to further clarify the reviewer's concerns.

In particular, following the reviewer's suggestions, we have (i) conducted additional experiments in other source-free scenarios, including the SFDA task (in Table 1) and the continual TTA setting (in Table 2), to further demonstrate the versatility of our AEA; (ii) clarified our TTA setup and expression of the equations in our paper. We have also included the raised points in our revised manuscript accordingly.

If the reviewer has any remaining concerns or suggestions, we would be grateful for the opportunity to respond further. Thank you for your valuable feedback and suggestions.

Motivation of our LCS loss

Our LCS loss is necessary to address the directional aspect of energy alignment, particularly concerning the class-wise adjustment of logit elements. Specifically, our SFEA loss alone minimizes the free energy of the target samples, aligning the energy between the source and target domains in terms of the magnitude of energy levels. That is, it increases the overall magnitude of the logits across all classes rather than considering the relativity of each logit element (i.e., class-wise correlation). On the other hand, our LCS loss additionally accounts for class-wise correlation by increasing the logit elements of classes with high correlation together. This allows each target sample to align with the well-trained source knowledge in a class-wise manner, leading to directionally improved energy alignment.

 

To support our claim, in Fig. 9 of Appendix, we provide sample-wise visualizations (similar to Figs. 1-(a) and (b)) with and without our LCS loss. In the results, while our SFEA loss effectively reduces the energy gap, a number of target samples are still not clearly separated according to their true classes. In contrast, by incorporating our LCS loss, the target samples align more successfully with the source samples in a class-wise manner, guiding the direction of energy alignment in a logit space. We have supplemented these points in Sec. A.4 of the Appendix, and we appreciate your comments.

Energy gap reduction for better adaptation in TTA

We appreciate the reviewer for this comment. We note that TTA is one of the distribution shift problems where the source and target domain distributions are different. As we figured out in Fig. 1, this domain disparity causes the energy gap, especially pronounced in the early-stage of batch arrivals. Although there are many methods to address the problem of distribution shift, our energy gap reduction can mitigate the domain disparity by promoting feature alignment between the source and target domains (as we discussed in Fig. 7). Therefore, our adaptive energy alignment approach can lead to better and accelerated adaptation in TTA, compared to earlier methods that handle distribution shifts without considering energy gap reduction. We have supplemented this point in Sec. 4.3 and Sec. A.8 of our revised manuscript.

Regarding performance improvements

First, we respectfully maintain that our AEA's performance improvement on average is meaningful. In fact, there are a few notable papers that reported similar types of gains at the time of publication relative to prior works [1-2]. For example, regarding CIFAR10-C, [1] shows an improvement of approximately 0.8% over TENT, whereas our AEA achieves a gain of 4.5% over TENT. But more importantly, though, we would like to stress that the key advantage of our AEA is its ability to accelerate adaptation, as demonstrated in Fig. 5. That is, our proposed method is preferable in practical scenarios where the number of online batches is small, leading to competent early-stage performance in TTA. Also, since our AEA only modifies the loss function, it introduces merely negligible extra delay/resources, which provides a practical advantage.

 

[1] Niu et al. Efficient test-time model adaptation without forgetting. ICML’22.

[2] Goyal et al. Test time adaptation via conjugate pseudo-labels. NeurIPS’22.

Dear authors,

I would appreciate your responses. I have read your rebuttal and I have the following additional remarks.

The impact of the idea.

I pretty much agree with them that the proposed energy alignment is very similar to the earlier works[1,2]. And the LCS loss is no further insight for the readers. It is not ready for ICLR in its current version.

About additional results.

I really appreciate that so many experiments are added quickly.

Ref: [1] Energy-based out-of-distribution detection. [2] Active learning for domain adaptation: An energy-based approach.

Connection between the EM loss and our AEA

$$H(x; \theta) +L_{SFEA}(x, \theta)= \sum_{j=1}^{K} p_\theta(y_j|x)E_{\theta}(x, y_j) - E_{\theta}(x) + \log(1+\exp(E_{\theta}(x)-\hat{E}_{s}))$$

Thank you for bringing up an important point. To provide a clear explanation, we present the summation of the EM loss and our SFEA loss above. First, in the EM loss, the first term encourages a reduction in energy, while the second term conversely encourages an increase in energy, resulting in conflicting roles for these two energy-based terms. Consequently, the EM loss alone has inherent limitations in sufficiently reducing the energy gap. To address these limitations, our SFEA loss (the third term) explicitly minimizes target energy, partially offsetting the free-energy maximization term (the second term) of the EM loss. This is why our method resolves the conflict and achieves better energy gap reduction compared to using the EM loss alone.

 

Also, we agree with the reviewer’s point regarding the necessity of the free-energy maximization term, as it appropriately regularizes the adaptation after the energy gap has been sufficiently reduced. However, in the early-stage of batch arrivals where the energy gap between source and target domains is pronounced, this term may restrict sufficient reduction of the energy gap (as observed in Fig. 1-(c) of the main manuscript), which is the main motivation of introducing our SFEA loss. Our SFEA loss is proposed to adaptively minimize target energy over time (strongly in the initial batches and more weakly later on), thereby reducing the energy gap more quickly. In turn, our method is designed to complement the limitations of the EM loss while leveraging its advantages in TTA (rather than replacing it), which is why we incorporate EM loss in our final objective. We clarified these points in the revised manuscript (in line 251~253 and Sec. A.2).

Energy gap reduction in the early batches

We appreciate the reviewer for this comment. In the results, at the beginning of the batch arrivals (i.e., ≤10 batches), the source pre-trained model has not been updated sufficiently to the target domain, which inevitably leads to minimal differences. Nevertheless, our AEA reduces the energy gap more than 6.4% compared to EM loss after only the initial 25 batches, which in turn leads to successful energy gap reduction in subsequent batches. Therefore, in TTA, it is important to reduce the energy gap in the early batches, and our adaptive energy alignment approach is effective in achieving this goal.

Ablation study on the choice of loss function

Thank you for your constructive suggestion. In table 1, we conduct an ablation study on the choice of loss function, comparing the hinge loss to our SFEA loss. Specifically, we report the results with varying $\lambda_1$, a coefficient of $\mathcal{L}_{SFEA}$, to evaluate hyperparameter sensitivity. The results demonstrate that our loss is less sensitive to hyperparameters than hinge loss, achieving better performance with marginal gains. Our intuition is that these improvements are attributed to the smoothness of the softplus function around the near-zero region, containing more information useful for reducing the energy gap than the hinge loss near that region.

 

We would also like to note that, unlike previous energy alignment works that have access to the source domain, the source-free scenario in TTA does not allow for accurate estimation of source energy. Under this new setting, we have utilized the approximated source energy for energy alignment, wherein our smoothed loss design proves to be more feasible and leads to better performance compared to the hinge loss. We have supplemented this ablation study with details in Sec. A.6.

 

• Table 1. Ablation study on the choice of loss function. We report the average classification accuracy (%, ↑) on CIFAR10-C dataset. Experimental details are consistent with those in the paper.

$\lambda_1$	1.0	1.2	1.4	1.6	1.8	2.0
hinge	80.3	80.2	79.9	79.7	79.3	79.0
ours	80.5(+0.2)	80.7(+0.5)	80.6(+0.7)	80.2(+0.5)	79.9(+0.6)	79.8(+0.8)

Dear Reviewer aNYF,

Thank you for your valuable and detailed comments to improve our work. As the discussion period is coming to a close, we would like to kindly remind you of our response to address the reviewer’s concerns.

Specifically, we have clarified (i) the relationship between the EM loss and our AEA, (ii) the energy gap reduction in the early batches, (iii) the choice of loss function, (iv) our approximated source energy approach in source-free TTA scenario and (v) other details regarding the equation of EM loss decomposition, the caption of Fig. 3, and the experimental results. We have also supplemented the raised points in our revised manuscript accordingly.

We sincerely hope to provide further clarification and discuss any additional questions the reviewer may have. Once again, we greatly appreciate your time and effort.