Rethinking Entropy in Test-Time Adaptation: The Missing Piece from Energy Duality

We appreciate the valuable time and effort you took to provide constructive and positive feedback. We believe these clarifications strengthen our manuscript and look forward to favorable consideration.

 

W1&Q1. When entropy is close to 0, I would imagine that the entropy loss and TCC are the same; if so, what does TCC do that entropy doesn't? Would it be possible to get more intuitive/practical explanations for the ways that the energy and TCC objectives help complement entropy? Are there images that are difficult for entropy based TTA to classify, but not for ReTTA?

We view TCC as a simple yet effective way to push the model output from merely low entropy to truly zero entropy, something that EM alone cannot ensure. Intuitively, the motivation is straightforward since zero-entropy predictions often correspond to highly confident and accurate classifications, as shown in Figs. 1(b) and 1(c). When a prediction is already at zero entropy, neither EM nor TCC makes further updates. In such cases, SSM pulls the sample deeper into the high-density, in-distribution region that EM alone cannot reach. By doing so, we encourage adaptation toward in-distribution and likely correct samples. So, for most inputs with non-zero entropies, TCC remains essential: in practice, as Table 3 shows, coupling state-of-the-art EM with TCC yields dramatic gains of 7% or 20% on corruptions like Glass or Frost, and achieves over a 1% average improvement versus the EM method (our in-house test of TCC with Tent shows a 0.9% average gain, from 43.1% to 44.0%). We also find EM often fails on images where the object is tiny or occluded, whereas ReTTA succeeds. A comprehensive analysis will require new metrics, which we plan to analyze in-depth.

 

W2. full IN-C, or at least, levels 1, 3, along with 5 should be shown.

We observed that at severity level 1, all TTA methods achieved comparable average performance (with ReTTA only marginally superior), whereas as corruption severity increased (levels 3 and 5; see Table 1), the combination of energy-based and entropy-based optimizations became important for maintaining higher accuracy.

A. ImageNet-C Severity 1 (ResNet-50-BN)

B. ImageNet-C Severity 3 (ResNet-50-BN)

W2&Q2. Would it be possible to see ReTTA performance on other ImageNet-size OOD datasets?

We evaluated ReTTA under `mild’ settings on three additional ImageNet-scale OOD benchmarks. ImageNet-R (rendered images) exhibits a large domain gap compared to ImageNet, while ImageNetV2 overlaps the original ImageNet distribution, and ImageNet-S (sketches) consists of single-channel grayscale outlines. We were unable to include ImageNet-3DCC, as its download alone exceeded 3 days and more than 340 GB and stalled. Therefore, we substituted ImageNet-S instead.

A. ImageNet R

B. ImageNetV2

C. ImageNet S

On ImageNet-R, ReTTA achieves gains comparable to those on ImageNet-C. On ImageNetV2, where other methods sometimes experience a performance drop, ReTTA offsets that penalty, paralleling our findings in the text-classification experiments (Response to YKNW). On ImageNet-S, ReTTA maintains the top rank among baselines, although the limited grayscale information and then reduced modes of distribution likely constrain the impact of SSM.

 

W3&Q3. Would it be possible to see ReTTA performance in the lifelong setting seen in EATA?

Looking back at Lemma 3, we could see that SSM might also help prevent forgetting. In each update, the “negative phase” gently raises energy (reducing the likelihood) for samples from the source distribution, while the “positive phase” lowers energy on the new corrupted samples. Rather than canceling out in a single SGLD iteration, these forces may reach a balance that slightly broadens the source density to cover new data without erasing its original support.

Indeed, when comparing ReTTA + EATA in the standard TTA setting (Table 4), we observe a modest average gain of +0.2%, and in the lifelong protocol, an even larger boost of +0.3%. These results suggest that SSM and EATA can be complementary not only for adapting to distributional shifts but also for preserving source performance over time, as evidenced by the average accuracies of 74.0% and 73.9%, respectively.

 

Minor weaknesses

Thank you for your careful reading and helpful suggestions. To address your remaining concerns, we will (1) add brief descriptions of the “Label Shift” and “Mild” settings; (2) due to the volume of experiments, we will reserve a small holdout subset of ImageNet-C corruptions for exploring $\lambda_2$, and (3) update Table 3 to include the missing EATA and SAR entries.

Thank you for your valuable and encouraging feedback. We truly appreciate your recognition of our work. As suggested, we will include the discussed results, including those on OOD ImageNet scale benchmarks, in the final version to better demonstrate ReTTA’s effectiveness across diverse settings.

 

Sincerely,

The Authors

We appreciate the valuable time and effort you took to provide constructive and positive feedback. We believe these clarifications strengthen our manuscript and look forward to favorable consideration.

 

W1. In the standard TTA setting (Table 1), ReTTA’s improvements over strong baselines like DeYO are relatively small (+0.6% average). The empirical results does not show a very significant advantage of using ReTTA, especially on mild scenario.

Q1. Figure 1 is not very clear to me, where I can't see much difference between Figure 1 (b) and (c). Are the authors suggesting that ReTTA (c) provides a clear improvement over EM (b)? If so, it would be helpful to highlight or quantify this difference more explicitly. Additionally, for (d), it would improve clarity to include a legend or annotation explaining the meanings of K and Z, so the figure is more self-contained.

Thank you for your careful reading and observations. It is worth noting that even on mild shifts, leading TTA methods yield only modest gains. For instance, DeYO improves over EATA by +0.8% on average (Table 1), so ReTTA’s +0.6% over DeYO is almost in line with the state of the art. When applied to SAR, ReTTA yields a +2.7% net gain in accuracy (Table 4), corresponding to roughly 1,350 additional correct predictions out of 50,000. Although adaptation does flip a few accurate predictions to incorrect and vice versa, the dominant effect is this net gain of 1,350 samples. This improvement explains why Figs. 1(b) and 1(c) appear visually subtle. To clarify, we will annotate the figure with the number of samples corrected and add a legend for Fig. 1(d) (refer to the caption of Fig. 2 in the Appendix for an example).

 

W2. While the method is motivated by covariate shift, the experimental setup includes a label shift, making it unclear what shift assumptions ReTTA is designed for. Given the shift in both p(x) and p(y), it is hard to say if this is still covariate shift or label shift since it's hard to maintain p(y|x) or p(x|y) unchanged.

Although the experimental protocols include both mild (covariate) and label shifts, this diversity cannot alter ReTTA’s core motivation: adapting to covariate shift in $p(x)$. In the mild setting, corruptions modify $p(x)$ under a uniform label distribution $p(y)$, so the resulting shift in $p(x|y)$ remains interpretable as a covariate shift. In label-shift tests, we replace the uniform prior with a skewed distribution $q(y)$, yielding $q(x)=\sum_y p(x|y)q(y)$, which again manifests as a new covariate shift in the marginal $q(x)$. Thus, regardless of stress-testing under combined shifts, ReTTA’s design and motivation remain focused on adaptation for covariate shifts.

 

Q2. The paper claims ReTTA is a “scalable solution” to distribution shifts in Conclusion. What exactly does “scalable” mean here, in terms of data size, model size, or corruption severity? Is there any empirical evidence supporting this?

We intended to highlight ReTTA’s ability to adapt to diverse distribution shifts, made possible by its self-adjusting coefficient, since this mechanism enables the method to balance likelihood and entropy signals on the fly without the need for manual tuning across different distribution shifts. Now, in response to Reviewer GM3k, YKNW, and 34Nx, we have further evaluated ReTTA across ImageNet-C severities (1 and 3), various Out-of-Distribution domains (ImageNet-R, V2, S), and even text classification tasks. We believe that these results can also support our argument that ReTTA scales across different domain types, modalities, and shift intensities.

 

Q3. I have doubt on the impact of TCC loss. Since test-time data is corrupted, the top-1 prediction may be wrong in some cases. Using it as a pseudo-target could mislead the model, especially in low-confidence cases. Table 3 suggests that TCC offers limited benefit when used alone. Could you provide more analysis on when and why TCC is effective?

Thank you for raising this important point. Our motivation behind TCC is to amplify the effect of entropy minimization (EM) by explicitly targeting samples near truly zero entropy where correctness is more likely, as shown in our empirical observations. TCC offers a straightforward and effective method for pushing predictions into this regime. However, we also recognize that such aggressive guidance may break the conservative nature of EM and could be vulnerable to errors if applied indiscriminately.

To mitigate this, as shown in Algorithms 1–3, we employ strong entropy-based data filtering —a strategy widely used in state-of-the-art EM approaches —to ensure that TCC is applied only when confidence is sufficiently high. In this design, TCC is not intended to function alone, but as a complementary module within EM pipelines that already incorporate safety mechanisms.

Thus, Table 3 should not be interpreted as showing TCC’s limited effect in isolation, but rather as evidence of its effectiveness under proper safeguards. In fact, when combined with EM, TCC delivers dramatic improvements of +7% and +20% on corruptions like Glass and Frost, and an average gain of over +1% compared to the EM method.

We appreciate your valuable time and effort in providing constructive and positive feedback. We have clarified the remaining points on Q1-Q3 below and added responses and a preliminary case study in text to show how ReTTA can extend beyond images.

 

Q1. Generalization to other image datasets]

ReTTA’s three pillars, Entropy Minimization (EM), Sliced Score Matching (SSM), and Targeted Class Convergence (TCC), are fully modular and plug-and-play for image classification. In practice, once a model (e.g., CNN or ViT) is trained:

1. Entropy: We can compute the entropy loss $-H(p)$ on images and take a gradient step in model parameters.
2. SSM: We only need the pixel-space gradient, which any differentiable model naturally provides.
3. TCC: We then apply a cross-entropy push toward the most probable class.

Therefore, no extra parameters and no architectural tweaks are required to move from ImageNet-C to variants like ImageNetV2, ImageNet-R, ImageNet-S, or other image datasets. We covered evaluations on these ImageNet variant datasets in response to Reviewer 34Nx.

 

Q2 and Q3. Applicability to text or speech modalities

While pixel-space gradients make SSM straightforward for images, raw text and audio waveforms lack such a grid. Directly computing score and Hessian on token IDs or raw waveforms may not capture meaningful structure. Instead, one should:

1. Choose a representation (e.g., token embeddings, spectrogram frames).
2. Compute then the Jacobian and Hessian trace in that representation space.
3. Formulate SSM based on those derivatives.

To show how this works in text, we conducted an exploratory sentiment classification study on Amazon Polarity Reviews.

 

A. Data, Model, and Task

We evaluated ReTTA on a cross-domain text classification scenario using the Amazon Polarity Reviews dataset. The source domain was Books, and the target domain was Electronics. Each review carried a 1–5 star rating; we binarized these into negative (ratings 1–2 $\rightarrow$ label 0) and positive (ratings 4-5 $\rightarrow$ label 1), discarding neutral reviews with rating 3. At test time, we deployed a Books-pretrained model to classify Electronics reviews under the TTA paradigm, using DistilBERT [1] from HuggingFace.

 

B. Implementation of SSM in Embedding Space

To extend SSM to text, we treated each input as the token-embedding sequence $s\in \mathbb{R}^{l\times 768}$ ($l$=sequence length) produced by DistilBERT’s encoder. For each $s$, we computed the Jacobian $\nabla_{s}\log p(s)$ and Hessian $\nabla^2_{s}\log p(s)$ via differentiation with the energy-based models with respect to $s$, i.e., $p(s):=\exp(-E_\theta(s))/Z(\theta)$. We then sampled random projection vectors $v\in\mathbb{R}^{l\times 768}$ and minimized the SSM loss:

$\mathbb{E}_{p(v)}\left[ v^\top \nabla^2_s \log p(s) v + \frac{1}{2} || v^\top \nabla_s \log p(s) ||^2 \right]$.

 

C. Experimental settings for text classification

We used the AdamW with a base learning rate scaled linearly by batch size, specifically setting $2e-5\times(\text{batch size} / 32)$ and a batch size of 256. The training epoch was set to 3. For Books training, we retrained all transformer layers and the classifier heads, keeping embeddings frozen. For Electronics TTA, we only updated the LayerNorm affine weights in transformer layers. Sequences were truncated/padded to 128 tokens; Books were split 90%/10% train/validation. Given that we addressed a single source to target shift, we did not employ a self‑adjusting coefficient; instead, we fixed both $\lambda_1$ and $\lambda_2$ to 1.

 

D. Preliminary Experimental Results

The DistilBERT model achieved 97.21% accuracy on the Books (source) domain. Without adaptation, the performance drops to 92.99% on Electronics (target). The results imply that the positive/negative sentiments from Books to Electronics highly overlap.

• EM only (Tent) fell to 87.73%, showing that naive EM can be harmful under mild shifts.

• Data filtering (entropy <0.2), adaptation with only those examples, yielded a marginal gain (87.75%) over EM.

• EM with SSM recovered much of the gap, boosting adaptation accuracy to 92.31% on Electronics.

• EM with TCC improved to 87.76%, which is marginally better than EM alone.

• ReTTA (EM + SSM + TCC) achieved 92.28%, comparable to SSM only but markedly better than EM-only.

These findings suggest that when the target distribution is not too dispersed, such as in the case of Books (source) and Electronics (target), where distributions remain similar, naive EM tends to be limited or even counterproductive. In these scenarios, energy‑based optimization can offer a solution by enhancing local observability. While EM struggles to detect how the probability landscape shifts around individual test samples, lowering a sample’s energy via SSM pulls not only that sample but also nearby points in the space into lower‑energy (higher‑likelihood) regions. This broader effect may help fill the observable gaps that EM alone often misses and mitigates misclassifications. Therefore, when the distribution overlaps between source and target domains are significant, incorporating minimization of the energy may provide an opportunity to alleviate the penalties incurred by EM substantially.

 

This feedback helped us uncover that EM performs well under large shifts (e.g., ImageNet‑C) but underperforms when source and target domains are highly overlapped (e.g., text sentiment). This study allowed us to rethink entropy and energy again: by lowering energy, we can both improve adaptation performance and compensate for EM’s blind spots in overlap scenarios.

 

Minor comments

Thank you for your careful reading and helpful suggestions. We will add a description for Fig. 1d in its caption (refer to the caption of Fig. 2 in the Appendix for an example), bold the highest DeYO values in the accuracy tables, and clearly distinguish entropy minimization from energy minimization throughout the manuscript to improve clarity.

 

---

[1] Sanh et al., DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, NeurIPS Workshop, 2019

We appreciate your valuable time and effort in providing constructive feedback. We believe that clarifications in response to the concern raised enhance the quality of this manuscript and hope for more favorable consideration of our research.

 

W1. The Sliced Score Matching component introduces second-order Jacobian terms, yet the paper provides no comparison of runtime or memory consumption against standard entropy minimisation (EM) or energy-based TTA baselines such as TEA

We appreciate the reviewer’s careful feedback. While Tent (EM-only) incurs minimal overhead, energy-based approaches generally involve higher memory and runtime demands. For example, TEA, which applied only an energy‑based contrastive divergence (CD) loss, requires approximately 8 GB of memory and exhibits significant runtime due to its generative procedures. In comparison, our combined EM + SSM implementation uses around 10 GB, showing that while the energy‑based method is inherently more costly than EM, our SSM adds only a modest overhead. Notably, our method runs about 30% faster than TEA, thanks to optimized Jacobian/Hessian computations that outperform TEA’s generative CD updates on batch data.

Tested on: Intel(R) Xeon(R) Silver 4214 @ 2.20GHz CPU, a single NVIDIA TITAN RTX 24GB GPU

We emphasize that this work prioritizes algorithmic contributions as a foundation for future optimizations. As energy-based TTA continues to mature, we expect engineering advances—such as memory-efficient formulations, faster approximations, and improved system-level support—to follow naturally.

 

W2. The proposed Targeted Class Convergence loss adopts the current top-1 output as a pseudo-label, which could amplify this mis-calibration and accumulate errors under heavy domain shift; the manuscript neither analyses the phenomenon nor proposes safeguards

We did not intend TCC to serve as a standalone approach. Rather, we were motivated that EM alone could not guess a test sample’s neighborhood or accurately estimate its likelihood, so we first applied SSM to pull samples (and their neighbors) into high‑likelihood regions. We could then use TCC to sharpen class probabilities on these refined representations. By integrating our losses into state-of-the-art EM workflows, we also inherited an entropy-based data filtering step, adapting only to low-entropy (and thus highly reliable) samples, to naturally guard against error amplification. In this design, SSM provided the core likelihood safeguard, data filtering prevented error accumulation, and TCC delivered the final discriminative push. As Table 3 shows, coupling EM with entropy‑based data filtering alone yielded dramatic improvements of 7% or 20% on corruptions like Glass or Frost, and on average delivered over a 1% gain versus the EM method. Also, the full ReTTA with SSM produced an average increase of over 2.5% relative to the EM method. It confirms that these components work in concert for robust, strong adaptation performance.

 

W3. The influence of the $\lambda_1$ weight on SSM, as well as the dynamics of the self-adjusting coefficient, are not ablated, leaving their robustness and default settings unclear

We note that our self‑adjusting coefficient is exactly $\lambda_1$, which by design adapted automatically to any distribution shift—so a fixed‑value ablation was unnecessary. As shown in Figure 1, the bar plot tracked $\lambda_1$ across different corruption types and batch iterations, demonstrating how it dynamically scaled the SSM gradient without a manual option.

 

W4. While the method is evaluated on ViT-Base under severe ImageNet-C corruption, results for the same architecture in the mild setting are absent, limiting evidence of effectiveness across shift intensities.

To ensure fairness, we used the DeYO source code repository to run the full suite of 15 severity‑5 corruption evaluations on ViT‑Base (LayerNorm) ourselves. The complete results appear below. Even under this mild setting, ReTTA led on average, achieving the highest accuracy on 10 of the 15 corruptions and second‑best on the remaining five. This confirmed that, across different model architectures, under severe distribution shifts, the simultaneous contributions of energy minimization and discriminability were crucial. Due to time constraints, we could not extend this evaluation to all five severity levels; instead, we performed additional tests on ResNet‑50‑BN at severities 1 and 3 (for Reviewer 34Nx), which mirrored these performance trends and further validated ReTTA’s robustness across varying shift intensities. We observed that at lower severities, all TTA methods delivered similar average performance (with ReTTA only marginally better), whereas as severity increased, jointly minimizing energy became essential to sustain high accuracy.

We hope these clarifications have effectively addressed your concerns and provided a clearer understanding of our contributions.

Thank you for your valuable and encouraging feedback. We truly appreciate your recognition of our improvements and overall contribution. Also, we are glad that our responses helped address most of the concerns.

 

Sincerely,

The Authors