Thank you for your detailed and encouraging review. We appreciate your recognition of the clarity of the writing, the theoretical support for our variance-based perspective, and the effectiveness of our design under low-batch test-time scenarios. Below, we respond to your comments and provide clarifications where needed.

W1. Comparative results under varying batch sizes

Thank you for the suggestion! To address this concern, we conducted additional experiments comparing Mint with the most competitive baselines (CLIPArTT and WATT-S) under varying batch sizes $B = 1, 2, 5, 10, 20, 50$. The results show that Mint consistently outperforms these baselines, particularly in more constrained settings (e.g., batch size = 1, 2, 5, 10), where it demonstrates greater stability and a more pronounced performance advantage. These findings further highlight Mint's effectiveness and robustness across different batch sizes.

Table A: Accuracy, ViT-B/32 on CIFAR-10-C (CLIP: 59.0)

Table B: Accuracy, ViT-B/16 on CIFAR-100-C (CLIP: 35.8)

Table C: Accuracy, ViT-L/14 on ImageNet-C (CLIP: 39.6)

Q1. Batch size vs. accuracy

This is because Mint leverages both a mean accumulator and a gradient accumulator, which allow it to aggregate information from all previously seen test samples, not just the current batch. As a result, a smaller batch size does not necessarily imply "less data" for adaptation, and thus does not always lead to degraded performance. And when the batch size becomes sufficiently large, Mint's performance tends to stabilize, and the small variations observed are primarily due to randomness rather than systematic effects.

As shown in Figure 6, when either the mean accumulator or the gradient accumulator is removed, the resulting ablated variants fail to fully utilize information from prior batches. In these cases, performance drops more significantly as batch size decreases, highlighting the importance of both components in maintaining robustness across batch sizes.

Thank you for your thoughtful and encouraging feedback. We appreciate your recognition of our theoretical insights and efforts to address the challenges of small-batch TTA. Below, we respond to your concerns point by point.

W1. Idea of variance collapse

Thank you for the helpful feedback and references. While we agree feature collapse under low-resolution inputs has been previously studied, our work explores a broader set of 15 corruption types spanning noise, blur, weather, and digital transformations (e.g., shot noise, motion blur, snow), many fundamentally different from low-resolution. Importantly, our key contribution is not only observing variance collapse but quantifying it, providing theoretical insights, and designing a corresponding adaptation loss. We appreciate your suggestions on relevant literature and will include these references to better contextualize our contributions.

W2. Confusion in theoretical analysis

Thank you very much for carefully reading through our theoretical analysis. Below, we address your concerns and clarify the derivations:

a. Layer norm

We used a simplified form of LayerNorm for analytical tractability. The first $\text{normalize}(\cdot)$ in our derivation comes from a simplified but general form of LayerNorm. Let the input of LayerNorm to be $v_i = [v_{i1}, \cdots, v_{id}]^\top \in \mathbb{R}^d$, and the output be $u_i\in\mathbb{R}^d$. The standard LayerNorm can be written as $u_i = \frac{v_i - \bar{v}\_i}{\sqrt{\frac{1}{d}\sum_{j=1}^d(v_{ij} - \bar{v}_i)^2}} \odot w + b$

where $\bar{v}\_i=\frac{1}{d}\sum_{j=1}^d v_{ij}$. As mentioned in Line 563, for analytical simplicity, we omit the demeaning step and the bias term, which leads to the following form: $u_i = \frac{v_i}{\sqrt{\frac{1}{d}\sum_{j=1}^d v_{ij}^2}} \odot w$ This corresponds to the widely used RMSNorm [1], a common simplification of LayerNorm that retains the normalization effect. Factoring out the constant $\sqrt{1/d}$ and writing the result in terms of L2 normalization: $u_i = \sqrt{d} \cdot \text{normalize}(v_i) \odot w$ Finally, the second $\text{normalize}(\cdot)$ in our derivation is not part of LayerNorm or RMSNorm, but instead comes from the CLIP architecture (see Figure 3 in [2]), where the output image embeddings are L2 normalized before computing cosine similarity.

We will revise the text to make this simplification process clearer and avoid ambiguity.

[1] Root Mean Square Layer Normalization. NeurIPS 2019

[2] Learning Transferable Visual Models From Natural Language Supervision. ICLR 2021

b. Feature disentaglement

To clarify, we use concatenation to construct the feature and weight vectors, and $\odot$ denotes the Hadamard (element-wise) product, not the dot product. $v^{cls}$ and $w^{irr}$ applied to different subspaces and their Hadamard product is not well-defined.

This decomposition follows a common analytical framework used in prior works [3,4], and is mainly intended to help identify which part of the feature vector each component of $w$ operates on, making the theoretical results more interpretable. In a more general setting where $v_i$ undergoes a rotation, each dimension becomes a mixture of the original components, and the gradient on $w$ will be a rotated version of our current result in Theorem 3.2. While the essence of the conclusion remains, such formulations would significantly complicate the interpretation.

[3] A fine-grained analysis on distribution shift. ICLR 2022.

[4] Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors. ICLR 2024.

c. Inequality

We would like to clarify that the derivation in Appendix Line 576 does not use the Cauchy-Schwarz inequality. All expressions are exact equalities. The identity holds because we assume that $w^{cls}$ is initialized as an all-one vector. As a result, the Hadamard product $u\odot w^{cls}=u$ holds for any $u$, and no distributional assumption is involved.

d. Convergence of means

We would like to clarify that the expression on Line 580 does not treat the mean vector as L2-normalized. The normalization factor $Z(w, s)$ arises from the final normalization step in CLIP's image encoder, and applies to each individual feature vector $z_i$; it is not related to the sample average.

Specifically, the convergence $\bar z \xrightarrow{p} \mathbb{E}z_i$ follows from the Weak Law of Large Numbers, indicating that the empirical average converges to the expected value. Each feature vector is defined as: $z_i=\frac{1}{Z(w, s)} \cdot [v^{cls} \odot w^{cls}; v^{irr} \odot w^{irr}; \delta \odot w^{shift}; v^{noise} \odot w^{noise}].$ To compute the expectation, we note that $\mathbb{E} v^{cls} = 0$ due to class balance, $\mathbb{E} v^{irr} = 0,\mathbb{E} v^{noise} = 0$ due to the Rademacher distribution. Thus, we obtain: $\bar{z}\xrightarrow{p}\mathbb{E}z_i=\frac{1}{Z(w, s)} \cdot [0; 0; s \cdot \delta \odot w^{shift}; 0]$

e. B.4 Adaptation

“Cov” in Line 589 refers to the covariance operator. The first two lines follow directly from the definition of covariance $Cov(X,Y) = \mathbb{E}[X,Y] = \mathbb{E}[X] \mathbb{E}[Y]$. The third line follows from the Weak Law of Large Numbers.

f&g. Theorem 3.2

This refers to Equation 4, where the gradient of PL-inter variance with respect to $w^{shift}$ is negative. Since we maximize $\mathcal{V}\^{PL}\_{inter}$ via gradient ascent, the update takes the form: $w^{shift} \leftarrow w^{shift} + \eta \cdot \nabla_{w^{shift}} \mathcal{V}^{PL}_{inter}$ where $\eta$ is the learning rate. Because the gradient is negative, this update reduces the magnitude of $w^{shift}$, effectively suppressing the structured shift component.

Similarly, we show that the gradient with respect to $w^{cls}$ is positive, meaning the same gradient ascent step will increase $w^{cls}$, thus amplifying task-relevant features.

h. Equation reference

Thank you for the feedback. We appreciate the suggestion and will revise the text to more clearly reference the equations associated with each inference to improve readability and traceability. If there are specific instances that are considered unclear, we would greatly appreciate it if you could point them out, so we can address them more precisely in the revision.

i. Accuracy

Theorem 3.1 shows that GT-inter variance increases with corruption severity, indicating a lower proportion of task-relevant features in the overall representation. This implies a reduced signal-to-noise ratio, which typically leads to lower accuracy. We will clarify this connection in the text.

W3. Ablation study

a. Maximize inter variance

If we directly maximize the PL-inter variance as defined by the LHS of Equation 5, we need to remove the mean accumulator (otherwise, the objective has zero gradients with respect to the current batch). We maximize the objective of $\frac{1}{C_b} \sum_{c=1}^{C_b} \\| m_c - m \\|\_2\^2$ where we use batch statistics $m_c = \frac{\sum_{i \in \mathbb{B}\_b} \hat{y}\_{ic} z_i}{\sum_{i \in \mathbb{B}\_b} \hat{y}\_{ic}}, m = \frac{1}{\mathbb{B}\_b} \sum_{i \in \mathbb{B}\_b} z_i$ as approximations of $\tilde{z_c}$ and $\tilde{z}$. The results are shown below for different batch sizes.

Notice that this objective is numerically equivalent to the "gradient accumulator only" variant in Figure 6, and thus yields the same results in practice.

b. Other variations

We tried several variations, including updating the last block, the last MLP layer, and the patching layer of the image encoder, but found that updating all LayerNorm layers yields the best performance. This choice is also consistent with prior works such as CLIPArTT and WATT, which also adapt LayerNorm.

W4. Why TTA

Thank you for the comment. We would like to clarify the motivation and setup of our experiment setup.

• We follow the most standard TTA setting widely used in the literature [5,6,7,8]. Specifically, we do not train a model on each corruption. Instead, we use the same pretrained VLM, and perform TTA separately on each corruption type, using only unlabeled test samples. This per-corruption evaluation is a common and practical protocol designed to assess the average performance of TTA algorithms under different corruptions, in a controlled and interpretable manner.
• This setup does not assume the model “knows” the corruption types. During training, the model is never exposed to these corruptions. The corruptions are only encountered at test time, during adaptation, which is consistent with the goal of evaluating corruption-agnostic TTA methods.
• Regarding the statement “not training on it is purely a design choice”, this is not the case. Our focus is on the adaptation of pretrained VLMs, where we only have access to public checkpoints and no access to training data. This is a practical constraint, not an arbitrary design decision, and reflects real-world deployment conditions for foundation models.

[5] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models. CVPR 2024

[6] Efficient test-time adaptation of vision-language models. CVPR 2024

[7] WATT: weight average test time adaptation of CLIP. NeurIPS 2024

[8] Clipartt: Adaptation of CLIP to new domains at test time. WACV 2025

W5. Translation to the real world

We conduct experiment on Natural Distribution Shifts Benchmark (please refer to Table B in our response to Reviewer Cqv3). Mint remains effective when faced with real-world images.

Figure 3

In Figure 3, severity is encoded by the color intensity of each point, with darker shades indicating higher corruption severity levels.

Missing citation

We will include the WACV’25 paper in our revision and clarify its connection to our method.

For your reference, here is a short description from their github repo:

ImageNet-R(endition) contains art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game renditions of ImageNet classes.

ImageNet-R has renditions of 200 ImageNet classes resulting in 30,000 images.

And here is the table including the ImageNet-R dataset.

Table B: (Updated) Performance on Natural Distribution Shifts Benchmark (ViT-B/16)

Thank you for your follow-up. Below are the direct answers you requested:

Q: The results reported in the paper were from 1 single TTA being evaluated on all forms of noise? Yes or No?

A: If by “TTA” you refer to the algorithm and its hyperparameters: Yes. If you refer to the adapted model parameters after TTA, then No—each corruption type results in a separately adapted model, since TTA is performed independently on each stream.

Q: The results reported in the paper had different TTA being evaluated on all different noises? Yes or No?

A: If by “TTA” you mean the algorithm: No. We use the same algorithm and hyperparameters across all corruptions. If you mean the adapted model weights, then Yes, because adaptation is performed independently per corruption type.

---

For example, the high-resolution (>128×128) sketch will not cause feature collapse on these datasets, and hence the entire story or theoretical foundation of layer norm update will be invalidated.

Respectfully, we believe there is a misunderstanding here. Nowhere in our paper do we state or assume that feature collapse only happens at low resolution. The paper you referenced in your earlier comment discusses feature collapse in low-resolution settings, but our analysis shows that variance collapse can also occur in high-resolution corrupted images, such as those from corruptions in ImageNet-C. Does ImageNet image + frost decrease resolution? I don't think so. The phenomenon is not inherently resolution-dependent, but corruption-dependent, and more broadly, distribution-shift-dependent.

---

Finally, we would like to clarify one point for better mutual understanding:

You mentioned that you're “not questioning the methodology of TTA,” but rather “the use of TTA for robustness.” Also it seems like you do not accept the idea of doing unsupervised fine-tuned for zero-shot CLIP model. Could we confirm whether your view is that test-time adaptation methods in general should not be considered valid robustness techniques, and that models should instead be evaluated in a domain generalization setting with frozen parameters?

We ask this in good faith, as it would help us better understand your criteria and expectations for robustness evaluation.

Thank you again for your time and engagement.

I see the confusion here, finally. I'm not trying to be hostile/rude here; my sincere apologies if my message seemed that way.

• Take the example of Table 1. There are 15 columns corresponding to 15 noises. That means 15 times TTA was adapted.

• The semantics here are: TTA wasn't informed what noise it was going to be adapted to. However, once TTA is adapted to a certain noise, such as a Gaussian blur, it won't work for Motion Blur.
  Pro: The Model can blindly be adapted to any 1 particular type of noise.
  Con: The Model is sort of domain-aware.

• Super Resolution / Lr-TK0 :
  Pro: The Model can be fixed without being shown the domain of the dataset. True zero-shot
  Con: The Model needs to be told what noise it's going to be shown.

• Both methods are solving the same task, but semantics are different, i.e. caters to one particular type of noise. One model parameter tuned for one type of noise won't work on another type of noise, which brings them on the same level in terms of the tasks they are trying to achieve.

• Here, our perspective is fundamentally different, as I would keep them in the same category of problems. I leave it to the area chairs to decide after this.

---

---

• The 128x128 was an example. Unless we are talking about noise (motion blur, rain etc.) all of these constitute a domain shift rather than a distribution shift.
• A cat in ImageNet vs a sketch of a cat is a domain shift. Clear cat in ImageNet vs Cat in snow is a distribution shift. Feature collapse / Variance collapse occurs mostly on Distribution shift (cat in snow/rain etc.), not so much in the sketch of the cat or the video graphic rendition of the cat.

---

---

• test-time adaptation methods in general. The way I'm seeing it, if I'm already aware of the test set being already sorted or made conveniently organized. Then yes, I would strongly favor a fine-tuning method over TTA.
• Example BDD-100K dataset. Its test/train images are well sorted in different weather conditions: sandy weather, rain, fog, etc. The Mint seems to be a method to evaluate on BDD per weather class, just by not telling what weather class its adapting to. TTA would have proved really useful here if Mint had evaluated all the BDD test sets in an amalgam (mixture) of ALL Noises. That's purely a design choice for me. I would train a classifier on the training set and fine-tune my model / use SR for the noise-specific train set. Then evaluate per noise category on BDD-100K. For me both are solving the same task. Not using the train set is a design choice. Using TTA is a design choice. The only way it would be a different problem statement if the authors left TTA to adapt to the entire test set with all forms of noise. Then it's fair to say SR, LR-TK0 are not in the same category of problem, and comparison is not meaningful. I guess our perspective on this are different, and I guess its ok.

Thank you very much for your positive and encouraging feedback. We're pleased that you found the paper well-structured, the motivation clear, and the proposed phenomenon and method compelling. We truly appreciate your support and are happy to address any remaining points or clarifications below.

W1. Assumption of disentangled components

Thank you for raising this point. We would like to clarify that the disentanglement assumption is primarily a conceptual tool to make the theoretical analysis more interpretable and analytically tractable.

In fact, our analysis can be extended to a more general setting where each feature vector undergoes a rotation, such that each dimension becomes a mixture of the original components. While this generalization does not change the conclusion of Theorem 3.1, the gradient derivation in Theorem 3.2 would involve additional rotation terms, making the result harder to interpret while offering limited new insight.

That said, we acknowledge that in extreme scenarios, such as when corruption completely entangles class-relevant signals (e.g., classifying different noise patterns themselves), the disentanglement assumption may indeed fail. In these cases, adaptation methods relying on discriminative features become fundamentally ineffective, as meaningful class-specific signals no longer exist or cannot be distinguished from corruption.

W2. ImageNets benchmark

Thank you for the insightful suggestion. Following your recommendation, we computed the GT-total, GT-intra, and GT-inter variances on ImageNet, ImageNet-A, ImageNet-R, and ImageNet-Sketch using the ViT-B/16 backbone. While this comparison may not be as tightly controlled as our corruption-based experiments, due to differences in image content and number of classes (e.g., ImageNet and ImageNet-Sketch have 1,000 classes, while ImageNet-A and ImageNet-R contain only 200), it still provides valuable insights.

Table A: Variances on Natural Distribution Shifts Benchmark (ViT-B/16)

On ImageNet-Sketch, we observed a clear variance collapse pattern consistent with our corruption benchmarks: both GT-inter and GT-intra variances decrease. However, on ImageNet-A and ImageNet-R, we observed that GT-inter variance still decreases, but GT-intra variance increases instead.

That said, the consistent drop in GT-inter variance across all three datasets supports the potential applicability of Mint to a broader range of distribution shifts, as it directly targets inter-class variance. Therefore, we evaluated Mint on these datasets, and the results are as follows:

Table B: Performance on Natural Distribution Shifts Benchmark (ViT-B/16)

Notably, Mint also outperforms our strongest baselines (CLIPArTT, WATT-S) from the main paper on all three datasets, further supporting its effectiveness beyond corruption benchmarks.

W3. Neural collapse

Thank you for the insightful suggestion. We agree that the idea of maximizing PL-total variance while minimizing PL-intra shares a conceptual connection with neural collapse phenomena, which also involve the alignment of features and separation of class means. We will include a discussion of neural collapse in the revised version to better contextualize our work within this broader literature.

Q1. Variances

Thank you for the question. In Line 136 and Figure 3, we are referring to the GT- variances (e.g., $\mathcal{V}\^{GT}\_{inter}$, $\mathcal{V}\^{GT}\_{intra}$), not the PL-variances. As shown in Tables 4–6 in Appendix C.1, all $\mathcal{V}\^{GT}\_{inter}$, $\mathcal{V}\^{GT}\_{intra}$ and $\mathcal{V}\^{GT}\_{total}$ positively correlate with accuracy, i.e., they decrease as corruption severity increases, and so does performance.

This is due to the phenomenon we described in Line 173: “As a result, the image encoder tends to embed corruption-related patterns into the representation itself.” Consequently, embeddings from different classes become more similar, reducing both inter- and intra-class variance, and degrading performance.

Q2. Model and dataset combination

Thank you for the question. The chosen model-dataset combinations reflect practical usage scenarios, where more efficient, lightweight models (e.g., ViT-B/32) are often used for simpler tasks like CIFAR-10-C, while stronger models (e.g., ViT-L/14) are typically employed for more complex tasks such as ImageNet-C. This setup ensures that the evaluation remains realistic and meaningful across a range of difficulty levels.

Thank you for taking the time to review our work and for your thoughtful feedback. We appreciate your recognition of our theoretical framework and experimental results. We also acknowledge your concerns and value your suggestions. Below, we respond to each point in detail and clarify potential misunderstandings.

W1. Decomposition in Eq (5)

Thank you for the question. You are right that even if certain classes have no samples, we can still evaluate the left-hand side of Eq (5) with the mean accumulator defined in Eq (6). However, the key issue with this formulation is that the loss becomes disconnected from the current batch, making it non-differentiable with respect to the current features $\{z\_i\}$. This prevents backpropagation and gradient-based optimization. Our reformulation (right-hand side of Eq. 5) addresses this by preserving differentiability via the use of batch-level embeddings $\{z_i\}$, while still leveraging $\tilde{z}_c$ and $\tilde{z}$ as running estimates. This enables optimization using stochastic gradients and facilitates information sharing across batches.

W2. Clean images and ImageNet benchmarks

Thank you for the helpful suggestion. We have added results on clean datasets corresponding to the three corruption datasets (CIFAR-10, CIFAR-100, and ImageNet), as well as four widely-used natural distribution shift datasets: ImageNet-A, ImageNet-V2, ImageNet-R, and ImageNet-Sketch. The table below reports accuracy under the same no-augmentation setting as used in our main experiments. We compare against the strongest baselines from the paper, including CLIPArTT and WATT.

Table A: Performance on Clean Datasets

Table B: Performance on Natural Distribution Shifts Benchmark (ViT-B/16)

W3. Sensivity to optimization space

Thank you for the question. In our method, optimizing the image encoder is essential, as our loss is explicitly designed to address the variance collapse in image embeddings. The PL-inter variance objective is defined on image features and is not differentiable with respect to the text encoder or prompt, since we treat the model’s predictions as hard pseudo-labels. As a result, updating only the text prompt or text encoder yields zero gradients and cannot adapt the model.

W4. Time comparison

Thank you for the suggestion. By “top-performing algorithms,” we refer to CLIPArTT and WATT-S, which achieve 4.9% and 6.1% accuracy gain, respectively. Following your advice, we have added all previously omitted training-free baselines (Ensemble, Zero, VTE, DMN, TDA) to the runtime comparison in Table 3, ordered by accuracy. This provides a more comprehensive reference for efficiency comparison.

Table 3: (Updated) Comparison of computation time for one corruption on CIFAR-100-C.

Minor 1. Hyperparameter $K_{prior}$

Thank you for pointing this out. We agree that the current phrasing may be misleading. We will revise the sentence to avoid confusion.

Minor 2. Statement conflict

Thank you for the question. We did observe a simultaneous reduction in both GT-inter and GT-intra variances. Among them, GT-inter variance, which reflects the separation between class centers, is naturally desirable to be large. On the other hand, a decrease in GT-intra variance, which reflects within-class variation, is not necessarily harmful.

In practice, we found that GT-inter variance correlates more strongly with accuracy than GT-intra and GT-total (see Tables 4,5,6 in Appendix C.1). Therefore, in our method, we choose to maximize PL-inter variance (computed as PL-total minus PL-intra), without explicitly penalizing PL-intra further in the loss.

Thank you for your acknowledgement of our responses to W3 and W4. Below we provide further clarification.

W1. Decomposition in Eq (5)

We agree with you that the objective formulation you propose is differentiable. In fact, we also experimented with it during the early stages of our work. However, we observed a gradient scaling issue with this formulation: as adaptation progresses, the contribution of each individual sample to the running mean becomes smaller, leading to systematically smaller gradients in later stages. This makes subsequent updates less effective and weakens the overall adaptation process.

To illustrate, consider the case where the batch size is 1. Let the only test sample in the batch have feature $z_i$, with pseudo-label $c$. Define:

• $\tilde{z}\^{pre}$ as the global average feature before the update;
• $\tilde{z}\^{pre}\_c$ as the class-$c$ average feature before the update;
• $\tilde{z}\^{post} = \frac{K}{K + 1} \cdot \tilde{z}^{pre} + \frac{1}{K+1} \cdot z_i$ as the global average after the update;
• $\tilde{z}\^{post}\_c = \frac{K_c}{K_c + 1} \cdot \tilde{z}^{pre}_c + \frac{1}{K_c+1} \cdot z_i$ as the class-$c$ average after the update; where $K$ is the total number of seen samples, and $K_c$ is the number of samples with pseudo-label $c$.

Under the formulation you proposed, the objective becomes:

\mathcal{V}\_{inter}\^{PL} = \\| \tilde{z}^{post}_c - \tilde{z}^{post} \\|_2^2 = \left \\| \frac{K_c}{K_c + 1} \cdot \tilde{z}_c^{pre} + \frac{1}{K_c+1} \cdot z_i - \frac{K}{K + 1} \cdot \tilde{z}^{pre} - \frac{1}{K+1} \cdot z_i \right\\|_2^2

Treating $\tilde{z}\^{pre}$ and $\tilde{z}\^{pre}\_c$ as constants, the gradient with respect to $z_i$ becomes:

$$\nabla_{z_i} \mathcal{V}\_{inter}\^{PL} = 2\left(\frac{1}{K_c + 1} - \frac{1}{K + 1}\right) \cdot \left ( \frac{K_c}{K_c + 1} \cdot \tilde{z}^{pre} + \frac{1}{K_c+1} \cdot z_i - \frac{K}{K + 1} \cdot \tilde{z}^{pre} - \frac{1}{K+1} \cdot z_i \right) = 2\left(\frac{1}{K_c + 1} - \frac{1}{K + 1}\right) \cdot ( \tilde{z}^{post}_c - \tilde{z}^{post})$$

As $K$ and $K_c$ increase over time, this gradient term shrinks toward zero. Consequently, new test samples make increasingly negligible contributions to adaptation.

In contrast, our objective takes the form:

$$\mathcal{V}\_{inter}\^{PL} = \\| z_i - \tilde{z}^{post} \\|_2^2 - \\| z_i - \tilde{z}^{post}_c\\|_2^2$$

Treating $\tilde{z}^{post}$ and $\tilde{z}^{post}_c$ as constants, the gradient with respect to $z_i$ becomes:

$$2(z_i - \tilde{z}^{post}) - 2 (z_i - \tilde{z}^{post}_c) = 2(\tilde{z}^{post}_c - \tilde{z}^{post})$$

This gradient is independent of $K$ and $K_c$, making the update scale stable throughout the entire adaptation process and ensuring that each test sample contributes equally, regardless of when it arrives.

W2. Clean images and ImageNet benchmarks

Thank you for your suggestion. We are currently running additional online TTA baselines, including TDA and other relevant methods, on our evaluation benchmarks. We will share these results with you as soon as they are available.

Hi Reviewer Cqv3,

Thank you very much for this insightful question. We truly appreciate your engagement with the theoretical details—it has been very helpful in clarifying and improving the presentation of our paper.

We agree with your observation: the right-hand side of Eq.5 does indeed depend on $K_c$. However, a key point we would like to emphasize is that Eq.5 is still not the final objective actually optimized during gradient ascent, the actual objective function used in each update step is Eq.7. (This is also true for the batch size = 1 special case we provided earlier)

If we were to use the RHS of Eq.5 directly as the loss, the normalizer’s denominator would be $K_c + 1$ as you said, which effectively corresponds to computing the loss over the full test stream—but only updating based on the current sample. This would lead to a gradient scale decay problem, since the influence of each new sample diminishes over time.

In contrast, Eq.7 is computed over only the current batch, and its normalization term is independent of the total number of previously seen samples. This design ensures that the gradient scale remains stable across adaptation steps, which is crucial for effective test-time optimization.

Thank you for pointing out the typo! we have corrected it in this response.

We would also like to clarify a potential misunderstanding regarding the gradient derivation. If the assumptions about what is treated as differentiable vs. constant are not changed, then of course the resulting gradient remains the same. However, our derivation and yours differ precisely in which values require grad and which do not.

---

In your version, $\tilde{z}_c^{pre}$ and $\tilde{z}^{pre}$ is treated as constants, which leads to a scale-decreasing gradient with respect to $z_i$, potentially causing optimization issues.

---

In contrast, our version treats $\tilde{z}_c^{post}$ and $\tilde{z}^{post}$ as constants. As you noted, this means we "did not take into account the other terms involving $\tilde{z}^{post}$ when calculating the gradient"—but we emphasize that this is by design, not an oversight. This design choice is what ensures that the gradient magnitude is independent of the number of seen samples $K$, and thus remains stable throughout optimization. In this sense, what may appear to be a “missing term” is actually a feature, not a bug.

---

We also followed your suggestion and double-checked our code implementation. In practice, we first update the mean accumulators under torch.no_grad, so that both $\tilde{z}_c^{post}$ and $\tilde{z}^{post}$ are computed with requires_grad=False before computing the loss. This aligns exactly with the form of the RHS of Equation 5 (and more precisely, Equation 7), confirming that our implementation is consistent with the derivation.